Haptically-Enabled Co-Robotics for Underwater Tasks

ABSTRACT

Apparatus configured for operation in an underwater environment are provided. A device includes a camera and a transceiver. The camera can capture, within a predetermined interval of time, first light within a first frequency range of light and second light within a second frequency range of light. The camera can generate a first image based on the first light and a second image based on the second light. The first frequency range of light can differ from the second frequency range of light; for example, the first frequency range can include 420-450 nanometers (nm) and the second frequency range can include 830 nm. The transceiver can send the images and receive commands based on haptic feedback generated based on the first image and the second image. For example, an actuator can be configured to be controlled by one or more commands.

RELATED APPLICATIONS

The present application claims priority to U.S. Provisional PatentApplication No. 61/756,132 entitled “Methods and Systems for SixDegree-of-Freedom Haptic Interaction with Streaming Point Clouds”, filedJan. 24, 2013, U.S. Provisional Patent Application No. 61/764,908entitled “Methods for Underwater Haptic Rendering Using NontactSensors”, filed Feb. 14, 2013, and U.S. Provisional Patent ApplicationNo. 61/764,921 entitled “Virtual Fixtures for Subsea Technology”, filedFeb. 14, 2013, all of which are entirely incorporated by referenceherein for all purposes.

STATEMENT OF GOVERNMENT RIGHTS

This invention was made with government support under grant no. 0930930,awarded by the National Science Foundation and with support under grantMRSEED01-006 “Haptically-Enabled Co-Robotics for Remediation of MilitaryMunitions Underwater” for the Strategic Environmental Research andDevelopment Program (SERDP). The United States Government has certainrights in the invention.

BACKGROUND OF THE INVENTION

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Haptic rendering is the translation of forces in a virtual environmentto a physical device that can provide touch-based, a.k.a. haptic,feedback to a user of the haptic rendering device. Both impedance typeand admittance type haptic rendering devices are available.

To provide haptic feedback about the virtual environment, objects in thevirtual environment are often represented as a collection of polygons,such as triangles, that can be operated upon using a haptic renderingdevice. The haptic rendering device can be controlled using a “HapticInteraction Point” (HIP) in the virtual environment, which performs asimilar function for the haptic rendering device as a mouse pointer doesfor a computer mouse. Ideally, the HIP should not be able to penetratevirtual environment objects.

FIGS. 1A-1C depict a scenario 100 that illustrates a prior art techniqueof utilizing proxy 130 to control interactions between HIP 110 andpolygon 120 in a virtual environment. In scenario 100, HIP 110 starts asthe position shown as HIP 110 a in FIG. 1A, moves through position HIP110 b shown in FIG. 1B, and ends at position HIP 110 c shown in FIG. 1C.In this technique, HIP 110 and proxy 130 are connected by a simulatedspring not shown in the Figures.

In FIG. 1A, proxy 130, shown at position 130 a, is in “free motion”;e.g., proxy 130 a is not touching polygon 120. In FIG. 1B, proxy 130,shown at position 130 b, is “in contact” with polygon 120. In scenario100, while HIP 110 continues to move down from position 110 b intopolygon 120, proxy 130 is not permitted to enter into polygon 120. InFIG. 1C, proxy 130, shown at position 130 c, is still in contact with asurface of polygon 120 after HIP 110 has moved to position 110 c insideof polygon 120. As the distance increases between HIP 110 and proxy 130,the simulated spring exerts a proportionally larger force to draw HIP110 closer to proxy 130. FIG. 1C shows the simulated spring force asforce 140 exerted on HIP 110. As shown in FIG. 1C, force 140 is exertedin the direction of a normal of the surface in contact with the proxy130; e.g., a hypotenuse 122 of polygon 120.

FIG. 2 shows an example coordinate system 200 specifying six degrees offreedom for tool 210. Coordinate system 200 can be used for otherentities as well, such as HIP. A position of tool 210 can be specifiedas a point (x, y, z) in three-dimensional space specified usingcoordinate system 200. For example, the point can defined in terms ofthree axes, such as an X axis, a Y axis, and a Z axis. Then, theposition of tool 210 can be specified as a (x, y, z) coordinate, with x,y, and z respectively specifying an X-axis coordinate, a Y-axiscoordinate, and a Z-axis coordinate. If only position is taken intoaccount, tool 210 can be said to have three positional degrees offreedom, as each of x, y, and z can be specified to position tool 210.

Tool 210 can be rotated about each of the X axis, Y axis, and Z axis, asshown in FIG. 2. If only rotations are taken into account, tool 210 canbe said to have three rotational degrees of freedom, as rotations abouteach of x, y, and z can be specified to rotate (or orient) tool 210.Taking both positional and rotational degrees of freedom into account,tool 210 can have up to 6 degrees of freedom, as listed on FIG. 2. Thesesix degrees of freedom include respective degrees of freedom forselecting an X coordinate, a Y coordinate, a Z coordinate, an Xrotation, a Y rotation, and a Z rotation.

Techniques for six degree-of-freedom haptic rendering in virtualenvironments consisting of polygons and/or voxels (volume pixels) havebeen specified. These efforts are typically divided into directrendering- and virtual coupling methods where the latter can further besubdivided into penalty-, impulse- and constraint-based methods. Thesimplest 6-DOF haptic rendering method is the direct method, where thevirtual tool perfectly matches the configuration of the haptic renderingdevice. The force sent to user is directly based on the amount ofpenetration in the virtual environment. Unfortunately the direct methodsuffers from problems with “pop through”. Pop through is an artifactthat arises when the rendering algorithm erroneously penetrates a thinsurface.

In virtual coupling methods, a virtual coupling, or connection, betweenthe haptic rendering device and the virtual tool is utilized. In thismethod, the force on the haptic rendering device is simply calculated asa spring between the virtual tool, referred to also as ‘god-object’,‘proxy’ or ‘IHIP’, and the configuration of the haptic rendering device.6-DOF rendering methods using virtual couplings rely on rigid bodysimulations, since the virtual tool has to be simulated as a 6-DOFobject, as compared to 3-DOF rendering where the rotational componentcan be ignored. In penalty-based methods, the configuration (positionand rotation) of the virtual tool is calculated using penalty forcesbased on the tool's penetration depth into objects, similar to howpenalty-costs are used in traditional optimization. These penalty forcesare then integrated to produce the motion of the virtual tool. Thismethod results in a virtual tool that actually penetrates objects in theenvironment. Fortunately this penetration is typically very small.

For impulse-based dynamics methods, a virtual object is moved by aseries of impulses upon contact/collision (rather than forces based onpenetration depth). In constraint-based methods, the virtual tool movesinto contact with the environment but (ideally) never violatesconstraints imposed by the environment.

Other environments can be explored by robots, such as undersea, outerspace, and hazardous environments. In some of these environments, robotscan be controlled by human operators receiving video and/or audioinformation from the robot.

SUMMARY

In one aspect, a method is provided. A computing device receives firstdepth data about an environment. The computing device generates a firstplurality of points from the first depth data. The computing devicedetermines a virtual tool, where the virtual tool is specified in termsof a translation component for the virtual tool and a rotation componentfor the virtual tool. The computing device determines a first forcevector between the virtual tool and the first plurality of points. Thecomputing device sends a first indication of haptic feedback based onthe first force vector.

In another aspect, an article of manufacture is provided. The article ofmanufacture includes a physical and/or non-transitory computer-readablestorage medium storing instructions that, upon execution by a processorof a computing device, cause the computing device to perform functionsincluding: receiving first depth data about an environment; generating afirst plurality of points from the first depth data; determining avirtual tool specified in terms of a translation component for thevirtual tool and a rotation component for the virtual tool; determininga first force vector between the virtual tool and the first plurality ofpoints; and sending a first indication of haptic feedback based on thefirst force vector.

In yet another aspect, a computing device is provided. The computingdevice includes a processor and data storage. The data storage storesinstructions that, upon execution by the processor, cause the computingdevice to perform functions including: receiving first depth data aboutan environment; generating a first plurality of points from the firstdepth data; determining a virtual tool specified in terms of atranslation component for the virtual tool and a rotation component forthe virtual tool; determining a first force vector between the virtualtool and the first plurality of points; and sending a first indicationof haptic feedback based on the first force vector.

In one aspect, a method is provided. A computing device receives firstdepth data about an environment. The computing device generates a firstplurality of points from the first depth data. The computing devicedetermines a haptic interface point (HIP). The computing device definesa virtual fixture for the environment. The computing device determines afirst force vector between the HIP and the first plurality of points.The first force vector is based on the virtual fixture. The computingdevice sends a first indication of haptic feedback based on the firstforce vector.

In another aspect, an article of manufacture is provided. The article ofmanufacture includes a physical computer-readable storage medium. Thephysical computer-readable storage medium stores instructions that, uponexecution by a processor of the article of manufacture, cause thearticle of manufacture to perform functions. The functions include:receiving first depth data about an environment; generating a firstplurality of points from the first depth data; determining a HIP;defining a virtual fixture for the environment; determining a firstforce vector between the HIP and the first plurality of points, wherethe first force vector is based on the virtual fixture; and sending afirst indication of haptic feedback based on the first force vector.

In yet another aspect, a computing device is provided. The computingdevice includes a processor and data storage. The data storage storesinstructions that, upon execution by the processor, cause the computingdevice to perform functions. The functions include: receiving firstdepth data about an environment; generating a first plurality of pointsfrom the first depth data; determining a HIP; defining a virtual fixturefor the environment; determining a first force vector between the HIPand the first plurality of points, where the first force vector is basedon the virtual fixture; and sending a first indication of hapticfeedback based on the first force vector.

In one aspect, a device configured for operation in an underwaterenvironment is provided. The device includes a camera. The camera isconfigured to capture, within a predetermined interval of time, firstlight within a first frequency range of light in the underwaterenvironment and second light within a second frequency range of light inthe underwater environment. The camera is configured to generate a firstimage based on the first light and a second image based on the secondlight, where the first frequency range of light differs from the secondfrequency range of light. The camera includes a communication interface.The communication interface is configured at least to send at least thefirst image and the second image and to receive one or more commandsbased on haptic feedback with the haptic feedback generated based on thefirst image and the second image.

In another aspect, a method is provided. A camera captures, within apredetermined interval of time, first light within a first frequencyrange of light in an underwater environment and second light within asecond frequency range of light in the underwater environment. Thecamera generates a first image based on the first light and a secondimage based on the second light. The first frequency range of lightdiffers from the second frequency range of light. The camera sends atleast the first image and the second image from the camera

One advantage of this application is the ability to specify andproviding haptic feedback for application using a virtual tool specifiedusing six degrees of freedom interacting with objects in an environmentspecified using point data with depth information, such data for aplurality of points. For example, three degrees of freedom can specify aposition of the virtual tool in a three-dimensional space, and threedegrees of freedom can specify an orientation, or rotation, of thevirtual tool with respect to the three-dimensional space. An exampleapplication for using the virtual tool is a task performed by auser-controlled robot, where haptic feedback is given to the user duringremote control of the robot.

Another advantage of this application is that haptic feedback isgenerated at rapid rates based on depth data, such as a stream of frameswith depth information. As disclosed herein, the use of a stream offrames with depth information permits haptic rendering, or providinghaptic feedback, at a haptic rendering rate faster than aframe-reception rate. In some embodiments, a herein-described hapticrendering system can have a haptic rendering rate of at least 1000 Hz.

Another advantage of this application is that virtual fixtures can bedefined based on objects recognized in the environment. That is, as anobject changes within the environment, a virtual fixture associated withthe object can dynamically adjust to the changes of the object. Anotheradvantage is that virtual fixtures can be dynamically changed based onstatus of operations, such tasks listed and organized on a task list.One or more virtual fixtures can be associated with task(s) on the tasklist. Virtual fixtures associated with a task can change based on thestatus of the task. Thus, the virtual fixtures can change throughoutexecution of a task, and so guide completion of the task. Further, alevel of automation can be specified that controls the virtual fixtures,and so allows for full automation, partial automation, or no automation(manual control) for completing the task.

Another advantage of this application is providing a camera thatcaptures images and depth information underwater. The images can becaptured in one range of frequencies, such as within a blue-green rangeof visible light frequencies. The depth information can be captured in adifferent range of frequencies, such as in a near-infrared range offrequencies. The visible light can be captured and a visible light imagegenerated. At the same time, NIR radiation can be captured and an imagewith depth information generated. The visible light image and the imagewith depth information can be sent from the camera in succession, sothat both visible-light information and depth information for a sametime can be provided. The visible-light information and depthinformation can be used to generate haptic feedback for operatorscontrolling devices to perform underwater tasks.

Providing haptic feedback from underwater sensors to an operator at thesurface has the potential to transform subsea manipulation capability.Virtual fixtures will allow manipulators to precisely maneuver insensitive environments where contact should not be made with surroundingobjects or structures. Biological studies of animal colonies aroundhydrothermal vents could benefit greatly from such a capability—allowingscientists to carefully gather data with unprecedented resolution andproximity to sensitive organisms. Common tasks such as connector matingbetween instruments can be carried out very efficiently by creatingguidance fixture near male and female connectors. Thus, time, expense,and equipment can be saved, and the environment preserved using theherein-described haptic feedback techniques to perform underwater tasks.

BRIEF DESCRIPTION OF THE DRAWINGS

Various examples of particular embodiments are described herein withreference to the following drawings, wherein like numerals denote likeentities, in which:

FIGS. 1A-1C depict a scenario illustrating a prior art technique ofcontrolling interactions between a Haptic Interaction Point (HIP) and apolygon in a virtual environment;

FIG. 2 shows an example coordinate system specifying six degrees offreedom for a tool;

FIG. 3A is a diagram of a haptic rendering environment, in accordancewith an example embodiment;

FIG. 3B shows an image based on captured depth data and the same imageannotated with normal vectors, in accordance with an example embodiment;

FIG. 4A depicts a scenario of capturing depth data in an environment, inaccordance with an example embodiment;

FIG. 4B depicts another scenario of capturing depth data in theenvironment shown in FIG. 4A, in accordance with an example embodiment;

FIG. 4C depicts a virtual tool and example points of depth data capturedfrom an environment, in accordance with an example embodiment;

FIG. 5A depicts points derived from depth data and a forbidden region,in accordance with an example embodiment.

FIG. 5B depicts a virtual tool in accordance with an example embodiment;

FIG. 5C shows an example virtual tool whose movement toward a desiredposition is impeded by a forbidden region, in accordance with an exampleembodiment;

FIG. 5D shows an example virtual tool whose movement is in collisionwith a forbidden region, in accordance with an example embodiment;

FIG. 6 is an example gaming scenario with haptic rendering, inaccordance with an example embodiment;

FIG. 7 is an example scenario for a haptic rendering session in a remoteenvironment, in accordance with an example embodiment;

FIG. 8 is an example scenario for collaborative haptic renderingsessions in a remote environment, in accordance with an exampleembodiment;

FIG. 9 depicts a construction scenario, in accordance with an exampleembodiment.

FIG. 10 shows an underwater robot, in accordance with an exampleembodiment; and

FIG. 11A is a block diagram of a computing environment, in accordancewith an example embodiment;

FIG. 11B is a block diagram of an example computing device, inaccordance with an example embodiment;

FIG. 12 is a flowchart of a method, in accordance with an exampleembodiment;

FIG. 13 is a block diagram of an architecture for rendering adaptivevirtual fixtures, in accordance with an example embodiment;

FIG. 14 is a block diagram of a pipeline for parallel processing andgeneration of virtual fixtures, in accordance with an exampleembodiment;

FIG. 15 depicts an environment with a robot arm and multiple virtualfixtures, in accordance with an example embodiment;

FIG. 16 is a flowchart of another method, in accordance with an exampleembodiment;

FIG. 17 is a block diagram of a system for a proximate operator toperform tasks in a remote physical environment, in accordance with anexample embodiment;

FIG. 18A is a block diagram of a camera configured to provide images forunderwater haptic rendering, in accordance with an example embodiment;

FIG. 18B is an exploded view of a camera configured to provide imagesfor underwater haptic rendering, in accordance with an exampleembodiment;

FIG. 18C depicts a sensor system, in accordance with an exampleembodiment;

FIG. 19 is a near-infrared image of an object in clear water, inaccordance with an example embodiment;

FIG. 20A is a near-infrared image of an object in murky water, inaccordance with an example embodiment;

FIG. 20B is an image representing depth data related to the image ofFIG. 20A, in accordance with an example embodiment;

FIG. 21 shows a graph of light absorption in water versus lightwavelength and a graph of light absorption in water versus lightwavelength over a range of water temperatures, in accordance with anexample embodiment; and

FIG. 22 is a block diagram of a time-multiplexed system of near-infraredcameras configured to capture images of an environment, in accordancewith an example embodiment;

FIG. 23 is a block diagram of a frequency-multiplexed system ofnear-infrared cameras configured to capture images of an environment, inaccordance with an example embodiment;

FIG. 24A shows a structure of a metal sensor, in accordance with anexample embodiment;

FIG. 24B is a diagram of an interferometer-based metal detector, inaccordance with an example embodiment;

FIG. 24C is a diagram of another interferometer-based metal detector, inaccordance with an example embodiment;

FIGS. 25A, 25B, and 25C are each a diagram of a metal detection systemthat includes multiple metal detectors, in accordance with an exampleembodiment;

FIG. 26A depicts a sensor system using the metal detection system ofFIG. 25A, in accordance with an example embodiment;

FIG. 26B depicts another sensor system using the metal detection systemof FIG. 25A, in accordance with an example embodiment;

FIG. 27A is an image of example metal objects, in accordance with anexample embodiment;

FIG. 27B is a graph of an output signal from an interferometer-basedmetal detector while scanning for metal objects, in accordance with anexample embodiment;

FIG. 27C is an outline of metal objects determined based on the data inFIG. 27B, in accordance with an example embodiment;

FIG. 28 illustrates a scenario where a vessel detects and retrieves anunderwater object, in accordance with an example embodiment.

FIG. 29 is a flowchart of a method, in accordance with an exampleembodiment.

DETAILED DESCRIPTION Overview

Haptic interaction has traditionally been a purely virtual task. Butrecent advancements in depth sensing have made it possible to streampoint data with depth information in real-time and at low cost; e.g., byusing a depth-enabled camera such as the camera used by Kinect from theMicrosoft Corporation of Redmond, Wash. For example, haptic interactionwith 6-DOFs from streaming point data can enable applications that fullyutilize a 6-DOF haptic rendering device, such as the ‘delta.6’ hapticinterface from Force Dimension of Nyon, Switzerland. A 6-DOF hapticrendering method could be useful even though many haptic renderingdevices only support 3-DOF actuation (but has 6-DOF sensing), such asthe PHANTOM Omni® haptic rendering device from Sensable Inc. ofWilmington, Mass.

Example 6-DOF applications include situations for remotely touching amoving object, such as remotely petting a dog. For example, thedepth-enabled camera can capture depth images of a remote environmentwhere the dog is located. The depth images can be transmitted to a localcomputing device. The local computing device can generate a virtualthree-dimensional environment showing the depth images (in this example,representing the dog) along with a virtual tool. The virtual tool can becontrolled by a local user of the haptic rendering device connected tothe local computing device during interaction with the virtualenvironment. For example, the virtual tool can represent a 6-DOF hapticrendering device controlled by the local user and the virtual tool cantake the form of a virtual human hand in the virtual environment. Theuser can control the 6-DOF haptic rendering device to provide commandsto move the virtual tool; e.g., move the virtual human hand to pet thedog. In response, a robot located in the remote environment can receiveequivalent commands to those provided to the virtual tool; e.g., so thatthe robot can move and pet the dog.

When haptic interaction is combined with manual control of a robot (orrobotic end effector) in co-robotic tasks, such as telerobotic surgery,a 6-DOF capability could add versatility and precision to thisco-robotic interaction. For example, it has also recently been shownthat the techniques of haptic rendering with 3 DOFs can be useful intelerobotics for implementation of virtual fixtures. That is, the userreceives a force when the teleoperated robot is near collision with theenvironment.

In some embodiments, virtual fixtures related to point data can bespecified. Examples of virtual fixtures include forbidden-regionfixtures and guidance fixtures. A forbidden-region fixture can define anarea represented by the point data where the virtual tool is notpermitted to enter; i.e., the forbidden region. A guidance fixture candefine an area represented by the point data where the virtual tool ispermitted, and perhaps encouraged, to enter. Haptic feedback can beprovided once the virtual tool attempts entry of a virtual fixture.

Another example use of haptic feedback is robotic or teleroboticsurgery. In telerobotic surgery, a surgeon can be located at onelocation which can be remote from an operating room that holds thepatient. During the operation, the surgeon can view the patient usingdepth images provided by a depth-enabled camera in the operating roomand transmitted to a local computing device. The local computing devicecan receive the depth images, generate a virtual environment for thesurgeon, receive commands via a 6 DOF haptic rendering device to controla virtual tool and corresponding surgical robot in the operating room,and generate haptic feedback that the surgeon can use to better treatthe patient.

Haptic feedback can also be useful in remotely controlling explorationdevices operating in hostile, dangerous, and/or unsafe environments.Example exploration devices include undersea and outer space explorationvehicles, explosive location devices, chemical sniffing (e.g., drugs,gas leaks, toxic waste) mechanisms, exploration robots, and/or otherexploration devices. For example, using haptic feedback, includinghaptic feedback in forbidden regions, can provide additional informationabout an environment under exploration and/or protect the explorationdevice from entering a known dangerous, already-explored, or otherwiseforbidden region.

Example Environments for Haptic Rendering

FIG. 3A is a diagram of a haptic rendering environment 300, inaccordance with an example embodiment. Environment 300 includescomputing devices 320 a, 320 b connected via network 310. Computingdevice 320 a is in remote environment 340 and is additionally connectedto remote controlled device (RCD) 336, and depth-enabled (DE) cameras346 a-346 d. Remote controlled device 336 is connected to tool 314.Computing device 320 b is additionally connected to display 312 andhaptic feedback device 316.

In the example shown in FIG. 3A, each of depth-enabled cameras 346 a-346d is configured to capture depth data of remote controlled device 336and objects 342, 344 and provide the captured depth data to computingdevice 320 a. One example of depth data is a depth image having observedcolor and depth values. In some embodiments, each of depth-enabledcameras 346 a-346 d can be configured with an optical camera to captureimage data and an infrared camera to capture depth data.

The depth image can be received as a NR×NC matrix of pixels, with eachpixel having 24 RGB bits of color value information, including 8 bits ofdata for each of red, green and blue component colors. The depth valuescan be received as a NR×NC matrix, where each depth value has B_(d)bits. For one example depth-enabled camera, the Kinect camera usesNR=640, NC=480, and B_(d)=11 or 13, and so produces 640×480 matrices ofimage data and depth data, where each depth value has at least 11 bits.The Kinect camera can simultaneously provide both image and depth dataat a frame rate f_(c) of 30 Hz. The depth values can represent depthrelative to the camera ranging from Dmin to Dmax, with Dmin representinga minimum distance from the camera, e.g., 0 mm; and Dmax, e.g., 2048 or8192 mm. Color and/or depth values can have additional information, suchas device information about the camera capturing the depth image.

In some embodiments, computing device 320 a can be connected to more orfewer depth-enabled cameras than shown in FIG. 3A, while remainingconnected to at least one depth-enabled camera. In other embodiments, adepth-enabled camera can generate other types of depth data than depthimages; e.g., depth-only information, point clouds. In yet otherembodiments, devices other than depth-enabled cameras can provide depthdata; e.g., depth sensors using radar or another technique to determinedepth.

Computing device 320 a can use received depth data to generate virtualenvironment 330. In order to interpret the depth data, computing device320 a can transform the depth data into a collection of points, eachpoint specified in a Cartesian coordinate system in three dimensions;e.g., as an (x, y, z) point in coordinate system 200 discussed above inthe context of FIG. 2.

Each (x, y, z) point from the depth data can represent one point in a“point cloud” or collection of points in three-dimensional Cartesianspace; e.g., (x, y, z)ε

³. In other embodiments, other data structures than point clouds orother collections of points can be generated from depth data. Aftergenerating the collection of points, computing device 320 a can rendervirtual environment 330 as images and/or as a three dimensionalvisualization. FIG. 3A shows virtual environment 330 including virtualobjects (VOs) 332 v, virtual valve 334 v, virtual remote controlleddevice 336 v, and a representative virtual tool 314 v.

Computing device 320 a and remote environment 340 can be physicallydistant from computing device 320 b. In some scenarios, remoteenvironment 340 can be physically near or in the same environment as anenvironment around computing device 320 b. In particular, of thesescenarios not shown in the Figures, one computing device can provide theherein-described functionality of computing devices 320 a and 320 b.

Computing device 320 a can generate force vectors related to tool 314and send indications of haptic feedback to computing device 320 b. Uponreception of the indications of haptic feedback, computing device 320 bcan utilize haptic interface device 316 to generate the haptic feedback.Additionally, computing device 320 a and/or 320 b can generatevisualization 318 with virtual object 332 v, virtual valve 334 v, and/orvirtual tool 314 v. As also shown in FIG. 3A, visualization 318 caninclude an indication of virtual tool 314 v which can correspond tovirtual tool 314 v in virtual environment 330 and/or tool 314 in remoteenvironment 340.

Haptic interface device 316 can be a controllable mechanism configuredto receive indications of haptic feedback and provide the indicatedhaptic feedback based on the indications. Example haptic interfacedevices 316 include, but are not limited to a delta.6 haptic interfacefrom Force Dimension, a PHANTOM Omni® Haptic Device from Sensable Inc.,other haptic devices, other haptic interfaces, haptic gloves, tactiledisplays, devices configured at least in part to provide haptic feedbacksuch as laptop computers, desktop computers, mobile telephones, hapticsuits, and/or other devices.

As haptic interface device 316 is moved, indication(s) of movement ofhaptic interface device 316 can be generated and sent from computingdevice 320 b, such as to computing device 320 a via network 310. Uponreception of the indication(s) of movement, computing device 320 a canupdate a position of virtual tool 314 v. Also or instead, computingdevice 320 a can send control signals to change movement and/orrotation; e.g., change speed, direction, acceleration, pitch, yaw, roll,or to stop) to remote controlled device 336 to move tool 314accordingly. In other embodiments, virtual tool 314 v can represent aposition of tool(s), sensor(s), and/or other device(s) on remotecontrolled device 336 other than tool 314, and by sending controlsignals remote controlled device 336, the tool(s), sensor(s), and/orother device(s) can be moved.

As depth data of remote environment 340 are captured, the captured depthdata can correspond to images and points showing movement and/orrotation of remote controlled device 336 and/or tool 314 and thusshowing movement and/or rotation of virtual tool 314 v. In someembodiments, remote controlled device 336 v and/or virtual tool 314 vcan be moved within virtual environment 330 based on the indication(s)of movement/rotation instead of or as well as based on captured imageand depth data.

An Algorithm for Moving a 6-DOF Virtual Tool

An algorithm is described for haptic interaction, which can beclassified as a constraint-based virtual coupling method. The hapticinteraction algorithm can support real-time haptic interaction withdiscontinuous streaming point data derived from depth sensors byiteratively resolving collisions for each received set of streamingpoint data; e.g., a received depth image.

The haptic interaction can occur between an arbitrary voxelized virtualtool controlled by a haptic rendering device and an environmentrepresented using streaming point data with depth information derivedfrom a depth sensor. A user of a haptic interface can direct a virtualtool to interact with both static point data for real objects anddynamic point data captured in real-time. The haptic interaction methodcan provide realistic forces/force feedback for a six degree-of-freedomhaptic interaction by performing a rigid body simulation of the virtualtool at a very high rate; in some embodiments, a haptic rendering rateof at least 1000 Hz can be supported.

The point data can be represented by one or more depth imagesrepresenting a set of fixed and infinitely stiff points in space. Eachdepth image can be filtered and surface normals for objects representedas points in the depth image can be calculated in real-time. Directinteraction between the virtual tool and the point data can be performedwithout first converting the point data to an intermediate datastructure to speed rendering.

A quasi-static (at rest) simulation can enable matching of the positionand orientation between the haptic rendering device and the virtualtool; i.e., a 6 DOF configuration. Collision between the virtual tooland the virtual environment can be detected. In the virtual environment,the virtual tool can be in one of the following three states: freemotion, in contact, or in collision. The free-motion state occurs whenno points based on depth information about the environment constrain thevirtual tool. The in-contact state occurs when there are points based ondepth information about the environment on the boundary of, but notpenetrating, the virtual tool. The in-collision state occurs when thereare points based on depth information about the environment thatpenetrate the virtual tool.

The quasi-static simulation can involve moving the virtual tool at eachof a number of time steps to enable the virtual tool to contact and thenbounce off a surface represented contact point(s) in the point data. Thevirtual tool is be simulated as an object at rest at the beginning of atime step. Then, the state of the virtual tool can be determined. If thevirtual tool does not collide with points in the environment, then thevirtual tool is in free motion; e.g., the virtual tool is in thefree-motion state. If a collision is detected between points in theenvironment and the virtual tool, then the virtual tool can either be incontact; e.g., the virtual tool is in the in-contact state, or thevirtual tool can be in collision; e.g., the virtual tool is in thein-collision state.

When the virtual tool is in contact, the points in the environment incontact with the virtual tool can form motion constraints for thevirtual tool. These constraints can be used to compute a constrainedmotion that will move the virtual tool towards the configuration of thehaptic rendering device without violating any contact constraints. Whenthe virtual tool is in collision, the collision can be resolved bymoving the virtual tool away from the set of collision constraints.

Periodically or otherwise, new information about the virtual environmentcan be received. For example, when a new depth data, such as a depthimage, depth frame, or other data including depth information iscaptured, the new depth data can be transformed into a point cloud orother structure representing points in the environment, where everypoint in Cartesian space corresponds to a pixel. The point cloud can befiltered; e.g., using a bilateral filter and a surface normal iscalculated for every point in the filtered point cloud. This surfacenormal calculation can lead to collision constraint(s) that point in thecorrect direction for a surface, regardless of which direction is usedto approach the surface. After the state of the virtual tool isdetermined, the force on the haptic rendering device is calculated as afunction of the kinetic distance between the configuration of thevirtual tool and the haptic rendering device.

Table 1 below includes example pseudo-code describing an example hapticinteraction algorithm regarding moving a virtual tool in a virtualenvironment.

TABLE 1 0001 done = false; 0002 WHILE (done == FALSE) DO 0003  //process depth data Din — see “Real-Time Processing of Streaming DepthData”  section. 0004  IF (new depth data available) THEN 0005   Receivedepth data Din from capture device, with Din including depth informationfor 0006    each of a number of pixels captured by captured device 0007  FOR each pixel P in Din DO 0008    Let Coords(P) = Cartesiancoordinates for P; 0009    Filter depth values of P; 0010    LetNormal(P) = normal vector for point P pointed toward capture device;0011   END FOR 0012  END IF 0013 0014  // determine motion constraints —see “Collision Detection with the Virtual Tool” section 0015  // assumeno points of Pin are in contact with VT 0016  Let VOX = voxelization ofvirtual tool VT; 0017  Let BB = bounding box of projection of VT ontoDin; 0018  Let state = free-motion; 0019  Let points_of_contact = NULL;0020  render_virtual_environment(Din, VT); // render the virtualenvironment with VT 0021 0022  // now, find out if any points are incontact with VT 0023  // if point is within bounding box, then see ifpixel corresponds to point within 0024  // voxelization VOX of VT. Ifpoint in VOX, then point is in contact or collides with VT 0025 0026 FOR each pixel P in Din DO 0027   IF (Coords(P) is within BB) THEN 0028   Let P’ = conversion of Coords(P) into coordinates used for VOX; 0029   Query VOX to determine if P' is a point within boundary of VOX; 0030   IF (P’ is within boundary of VOX) THEN 0031     Add P topoints_of_contact; 0032    END IF 0033   END IF 0034  END FOR 0035 0036 // if there are any points in contact, see if any points penetrate VT.If point penetrates VT, 0037  // then point is in collision with VT.Otherwise, VT is in-contact (just touching) the point. 0038 0039  IF(points_of_contact != NULL) THEN 0040   Let state = in-contact 0041  FOR each point PC in points of contact DO 0042    IF (penetrationdepth of PC into VT > 0) THEN 0043     Let state = in-collision 0044   END IF 0045   END FOR 0046  END IF 0047 0048  // move VT based onstate — see “Finding a Constrained Motion while In Contact” 0049  // and“Resolving Collisions” sections below 0050 0051 initialize_matrix(Matrix_J); // create/zero out Matrix_J as necessary0052 0053  IF (state ==free-motion) THEN // no constraints on moving VT0054   Move VT according to unconstrained acceleration of Equation (1)below 0055  ELSE 0056   // points in contact/collision add constraints0057   FOR each point PC in points_of_contact DO 0058    Determineconstraint(PC) using Equation (2) below 0059    Add constraint(PC) toMatrix_J; 0060   END FOR 0061 0062   // move VT based on state andconstraints 0063   IF (state == in-contact) THEN 0064    Move VTaccording to constrained acceleration (see Equations (3) and (4)) 0065   using nearest point algorithm; 0066   ELSE // state == in-collision0067    Move VT according to constrained acceleration using Equations(6) and (7) 0068   END IF 0069  END IF 0070 0071  // apply force tohaptic rendering device — see “Calculating the Force” section 0072  LetF = force determined by Equation (8) 0073  Send indication of F toprovide haptic feedback 0074 0075  // determine if we can terminatealgorithm 0076  done = are_we_done_yet( ); // or similartechnique/function 0077 END WHILE

The algorithm shown in Table 1 is specified as mainly as a loop betweenlines 0002 and 0077 that can iterate one or more times until a variabledone is TRUE. A loop iteration can begin at line 0005 by determining ifnew depth data Din is available. If so, the new depth data Din isprocessed at lines 0006-0013, and as further discussed in the “Real-TimeProcessing of Streaming Depth Data” section below.

As further discussed in “Collision Detection with the Virtual Tool”section below, lines 0014-0020 involve generating a voxelization VOX ofa virtual tool VT and a bounding box BB of a projection of virtual toolVT into a space for Din, as well as initialing a state variable,representing a state of VT, to “free-motion” and a points_of_contactlist to NULL (no points in list). Also, the virtual environment isrendered based on depth data Din and virtual tool VT. At lines0021-0034, a determination is made whether any points represented by thedepth data Din are in contact with virtual tool VT. At lines 0035-0046,the state of VT is determined: (a) if no points of Din are in contactwith VT, the state of VT remains set as “free-motion”, (b) if a point ofDIN is in contact with VT and if none of the in-contact points in thepoints_of_contact list has penetrated VT, the state of VT is set to“in-contact”, (c) else, at least one point has penetrated VT and so thestate of VT is set to “in-collision”.

At lines 0047-0069, virtual tool VT is then moved based on VT's state,which is discussed below in the “Finding a Constrained Motion while InContact” and “Resolving Collisions” sections below. If VT is in a stateof free-motion (VT is not in contact with any points of Din and nointerpenetrations), the movement is based on unconstrained accelerationspecified using Equation (1) below. That is, the virtual tool can bemoved in the direction of the unconstrained acceleration until firstcontact occurs. In some embodiments, the virtual tool can be moved insmall discrete steps to ensure that no contact is missed.

Otherwise, VT is moved according to constrained acceleration specifiedusing Equations (2)-(7) below. In some embodiments not shown in Table 1above, bisection can be applied at first contact to further refine thepoint(s) of contact. In other embodiments not shown in Table 1 above,the constrained acceleration can be applied in small discrete steps,where the magnitude per step is limited to a predetermined amount. Thevirtual tool can be moved in small discrete steps such that no featureis missed (as for the case when the virtual tool is in free motion).This procedure of movement using small discrete steps can be repeateduntil collision is resolved. In particular of these embodiments,bisection can be applied again until the collision is resolved.

At lines 0070-0073, force feedback is calculated and provided to a userof a haptic tool operating VT, discussed below in the “Calculating theForce” section. At the end of the loop iteration on lines 0074-0077, adetermination is made as to whether the algorithm should end or not. Ifthe algorithm should end, the done variable should be set to TRUE toterminate the loop, and therefore the algorithm. Otherwise, done shouldbe set to FALSE to lead to another loop iteration.

Real-Time Processing of Streaming Depth Data

FIG. 3B shows depth image 350 based on captured depth data and depthimage 350 annotated with normal vectors including normals 352, 354, 356,358, in accordance with an example embodiment. The upper image of FIG.3B shows depth image 350 of captured depth data representing a humanhand and arm.

As mentioned above, Cartesian coordinates can be determined for eachpixel in a depth image. For each pixel, color and depth values in thedepth image can be used to calculate Cartesian coordinates for acorresponding point. With the pixels in the depth image transformed topoints in a Cartesian coordinate system, the depth values can befiltered using a bilateral filter, where the points can be weightedusing a Gaussian kernel as a function of Euclidean distances in aCartesian frame. The points can be weighted to preserve depthdiscontinuities; i.e., a neighboring pixel only contributes to thefiltered value if its depth value is sufficiently close.

A normal vector can be calculated by fitting a plane to the points in asmall neighborhood around each point using a least squares technique.The neighboring points can be weighted using the smooth Wendlandweighting function (e.g., with radius rw). This weighted total leastsquares problem has a closed form solution that can be solved with a 3×3eigenvalue decomposition for every point. There can be two numericalsolutions to the least squares problem, where each solution is apossible normal. The normal can be selected as the numerical solutionpointing towards the depth sensor.

The lower image shown in FIG. 3B shows depth image 350 annotated bynormal vectors; e.g., normals 352, 354, 356, 358, where the normalvectors were calculated using the techniques discussed immediately abovefor each 1000^(th) point derived from depth image 350. In someembodiments, a normal vector can be calculated for every point in adepth image. In other embodiments, per-pixel data parallel work can beperformed by a computing device utilizing one or more graphicsprocessing units (GPUs). For example, the above-mentioned Cartesiantransformation, bilateral filtering, and/or least squaressolution/eigenvalue decomposition can be done for every pixel inparallel using OpenCL (Khronos Group Inc.) running on the GPU(s).

GPU performance can be optimized by copying several pixels at a time tolocal GPU memory. Then, the copied pixels can be processed in parallelby a work group of computational units; i.e., components of GPUs and/orentire GPUs. As local memory access by the computation units can be upto 2 orders of magnitude faster than remote memory access, performancecan be significantly speeded by use of local GPU memory. Local copyingcan be useful in filtering operations where many neighboring pixels areaccessed.

Collision Detection with the Virtual Tool

In order for the virtual tool to move with respect to the points derivedfrom the depth image, a quick collision detection method is needed. Insome embodiments, the virtual tool is voxelized. For example, the voxelscan be of a uniform size, such as 0.5 mm on a voxel side. Voxels can bestored in local memory using a simple scan-line technique.

To determine whether a point P collides with a virtual tool VT, thepoint P can be transformed to a corresponding point P′ in a frame ofreference for VT. Then, point P′ can be queried against the voxels usedto voxelize VT. In some cases, the query can be equivalent to a look upin a collision/no collision lookup table. In some embodiments, collisionlookup can be sped by checking for collision with points that are in theneighborhood of virtual tool VT. The neighborhood of VT can beapproximated using a two-dimensional bounding box placed around theprojection of the virtual tool onto the depth image. Then, a point Pncan be determined to be within the neighborhood of VT if a correspondingpixel Px to point Pn is a pixel within the bounding box projected on tothe depth image.

FIG. 4A depicts scenario 400 of capturing depth data in an environment410, in accordance with an example embodiment. The left side of FIG. 4Ashows an overhead view of environment 410, with depth enabled camera 412configured to capture depth data, such as depth images, of objects 414and 416 as well as tool 418. At a time T1 in scenario 400, tool 418 isin front of object 416 and to the left of object 414.

At time T1, depth enabled camera 412 captures depth image 420 ofenvironment 410. Depth image 420 is shown in the upper-right portion ofFIG. 4A, with object 414 on the left side of the image, object 416 inthe central portion of depth image 420, and a side view of tool 418shown in front of object 416. In scenario 400, after conversion ofpixels in depth image 420 to corresponding points in a Cartesiancoordinate system, each point is then checked for collision with avirtual tool representing tool 418.

As discussed above, a bounding box surrounding the image of tool 418 canprojected on to depth image 420. The lower-right portion of FIG. 4Ashows depth image 420 with projected bounding box 422 replacing aportion of depth image 420 depicting tool 418. Then, pointscorresponding to pixels of depth image 420 that are within bounding box422 can be queried for possible collision with tool 418. In scenario400, the corresponding points can be points captured from object 416,whose depth values would be larger than those of tool 418; i.e., object416 is behind tool 418. As such, no collisions would be detected betweentool 418 and points representing the rest of environment 410 in scenario400.

FIG. 4B depicts scenario 450 of capturing depth data in an environment410, in accordance with an example embodiment. The left side of FIG. 4Bshows an overhead view of environment 410, with depth enabled camera412, objects 414 and 416, and tool 418 as discussed above in the contextof FIG. 4A. At a time T2 in scenario 400, tool 418 is in front of object416 and just touching object 414.

At time T2, depth enabled camera 412 captures depth image 460 ofenvironment 410. Depth image 460 is shown in the upper-right portion ofFIG. 4B, with object 414 on the left side of the image, object 416 inthe central portion of depth image 460, and a side view of tool 418shown in front of object 416 and just touching object 414. In scenario450, after conversion of pixels in depth image 460 to correspondingpoints in a Cartesian coordinate system, each point is then checked forcollision with a virtual tool representing tool 418.

As discussed above, a bounding box surrounding the image of tool 418 canprojected on to depth image 460. The lower-right portion of FIG. 4Bshows depth image 460 with projected bounding box 462 replacing aportion of depth image 460 depicting tool 418. Then, pointscorresponding to pixels of depth image 460 that are within bounding box462 can be queried for possible collision with tool 418. In scenario450, some of the corresponding points can be points captured from object416 as discussed above in the context of FIG. 4A. Other of thecorresponding points can be points from object 414, such as points atthe left-most extent of bounding box 462 that just touches object 414.The depth values can be similar to those depth values representing aposition of tool 418; i.e., object 414 may be in contact with and/orpenetrated by tool 418. As such, collisions may be detected betweenobject 414 and tool 418 in scenario 450.

Finding a Constrained Motion while in Contact

Using this collision detection method, the virtual tool is moved towardsthe configuration of the haptic rendering device and stopped at thefirst point of contact. Gauss' least constraints principle can provide abasis for find a feasible movement for the virtual tool that will notviolate the contact constraint further.

The generalized unconstrained acceleration a_(gu) can be expressed usingEquation (1) below:

$\begin{matrix}{a_{gu} = \begin{pmatrix}a_{u} \\\alpha_{u}\end{pmatrix}} & (1)\end{matrix}$

where a_(u) is the unconstrained acceleration in the horizontal andvertical (X and Y) dimensions, and α_(u) is the unconstrainedacceleration in the depth (Z) dimension.

a_(gu) can be considered as an ideal unit step; e.g., a unit step thatcan be taken if the configuration of virtual tool VT were to match theconfiguration of the haptic rendering device. However if virtual tool VTis in contact, this ideal unit step cannot be taken. Each point ofcontact (as found by the collision detection) can introduce a linearconstraint on accelerations a and a specified by Equation (2):

n ^(T) a+(r×n)^(T) α≧d  (2)

For equation (2), n is the normal vector at the point of contact, r isthe vector from the center of mass to the point of contact, and d is thepenetration depth of the point of contact with d=0 when virtual tool VTis in contact with, but not penetrated by, the contact point and d<0when virtual tool VT is penetrated by the contact point.

Given a_(gu), a constrained acceleration a_(c) can be calculated whichminimizes the kinetic distance between the virtual tool and the hapticrendering device with respect to the linearized constraints. Moreformally, the constrained acceleration a_(c) can be found as

$\begin{matrix}{a_{c} = {\frac{1}{2}{\min\limits_{a_{g}}{( {a_{gu} - a_{g}} )^{T}{M( {a_{gu} - a_{g}} )}}}}} & (3)\end{matrix}$

subject to

Ja _(g)≧0  (4)

where M is the mass matrix for the virtual tool and J is a matrix ofconstraints formed by (2) for each point in contact.

Equations (3) and (4) represent a quadratic programming problem solvableusing Wilhelmsen's nearest point algorithm. The constrained accelerationa_(c) may be valid for only a small neighborhood around the currentconfiguration because of the linearized contact constraints J. Further,validity of a_(c) can be affected any movement by the virtual tool VTand/or changes in point positions introduced by new depth data(represented by Pin), as the movement can introduce new constraints onvirtual tool VT.

To illustrate the depth value d of Equation (2), FIG. 4C depicts virtualtool 470 and example points (Ps) 472 a, 474 a, and 476 a representingdepth data captured from an environment. For example, points 472 a, 474a, and 476 a can be points in a Cartesian coordinate system calculatedfrom pixels of a depth image, as discussed above. As mentioned above, anormal vector can be calculated by fitting a plane, or surface, to eachof points 472 a, 474 a, and 476 a a small neighborhood around eachpoint. For example, FIG. 4C shows an example surface (S) 472 bcalculated for point 472 a and corresponding normal (N) 472 c. FIG. 4Calso shows surfaces 474 b, 476 b and normals 474 c, 476 c for respectivepoints 474 a, 476 a.

The depth d for a point P with respect to virtual tool VT can bespecified as a distance from P to a portion of VT closest to P along thedirection of a normal vector N associated with P. For example, FIG. 4Cshows that point 472 a is a positive distance from a portion of virtualtool 470 closest to point 472 a in the direction of normal 472 c; thatis, the depth of point 472 a with respect to virtual tool 470 ispositive. As the depth of point 472 a is positive, virtual tool 470 canbe classified as in free motion, or unconstrained by, point 472 a.

As another example, FIG. 4C shows that point 474 a is just touching asurface of virtual tool 470. That is, point 474 a is a distance of 0units away from virtual tool 470 in the direction of normal 474 c, andso the depth of point 474 a is 0. As the depth of point 474 a is 0,virtual tool 470 can be classified as being in contact with point 474 a.FIG. 4C also shows that point 476 a is inside the surfaces of virtualtool 470. That is, point 476 a is a negative distance away from aclosest surface of virtual tool 470 in the direction of normal 476 c,and so the depth of point 474 c is negative. As the depth of point 476 ais negative, virtual tool 470 can be classified as being in collisionwith point 476 a. As such, virtual tool 470 can be considered to beconstrained by points 474 a and 476 a.

Resolving Collisions

One technique to resolve collisions between VT and points is based onthe use of interval arithmetic. Generally speaking, interval arithmeticoperates on ranges (or intervals) of values rather than specific values.For example, suppose bodies A and B are 6.5 feet apart. For intervalarithmetic, the relative position of A and B can be specified based on adistance interval; e.g., an interval between 6 to 8 feet.

As part of this collision-resolution technique, the step size to moveany part of the virtual tool can be constrained to be less than thesmallest size of voxels used to represent the virtual tool. For example,the step size can be chosen to be half the tool voxel size. Bounding thevirtual tool movement can ensure that no collisions are missed in asingle point image, or when static point data is used. However, whendynamic point data is used, bounding the virtual tool's movement doesnot hold after the point data updates, as the locations of pointsrepresented by a new point image or other representation of the pointdata can be arbitrary.

A quasi-static collision resolution technique can be used to resolvingcollisions between virtual tools and virtual environments represented bydynamic point data. Initially, virtual tool VT is at rest; e.g.,satisfies the condition specified by Equation (5):

a _(gu)=0  (5)

To move the virtual tool VT, the depth value d (discussed above in thecontext of Equation (2) and below in the context of FIG. 4C) has to benon-zero in order for the minimization to move the virtual tool out ofthe constraint. The value of d should ideally match the penetrationdepth at each point of collision. However, a planar constraint specifiedin Equation (2) may only be valid in a small neighborhood of virtualtool VT and since movements of virtual tool VT might introduce newconstraints.

An approximate solution to resolve collisions can be obtained byweighting the constraints equally and updating the configuration of thevirtual tool iteratively. This approach can be summarized as:

$\begin{matrix}{a_{c} = {\frac{1}{2}{\min\limits_{a_{g}}{a_{g}^{T}M\; a_{g}}}}} & (6)\end{matrix}$

subject to

Ja _(g)≧1  (7)

Note that the right hand side value “1” in Equation (7) can be chosen asany vector with identical elements. The choice of right hand side valuecan change the magnitude, but not the direction of the result fromEquation (6).

Calculating the Force

The force f sent to the haptic rendering device can be calculated as afunction of the difference between the virtual tool and haptic deviceconfiguration. For example, f can specified as f=(f_(T) f_(R))^(T), withf_(T)ε

³ being a translational component, and fRε

³ being a rotational component. Then, f_(T) can correspond to threedegrees of freedom for a position of virtual tool VT and f_(R) cancorrespond to three degrees of freedom for a rotation of virtual toolVT.

The force f can be calculated as:

f=KMa _(gu)  (8)

where K is a diagonal matrix containing spring constants between virtualtool VT and each contact point. For an example of spring constants, avirtual spring with a corresponding spring constant can be connectedbetween virtual tool VT and a contact point. As the distance between VTand the contact point increases, the virtual spring exerts aproportionally larger force to draw VT closer to the contact point. Thecalculated force can be exerted in the direction of the normal at thecontact point; i.e., corresponding to the virtual spring pulling VT andthe contact point together.

Example Implementation

As an example, the herein-described 6-DOF haptic rendering algorithm wasimplemented on a desktop computer (AMD Phantom II X6 with a Radeon HD6990 GPU) running Ubuntu 11.10. The force was calculated asynchronouslyat 1000 Hz in a separate thread. During typical interaction, thecollision detection algorithm ran at 15 kHz. Point images were filteredand normal vectors were calculated for every point using the GPU.

Realtime processing was achieved using a neighborhood of 9×9 points forfiltering as well as normal vector calculation. The position of thehaptic rendering device was both controlled automatically (for purposesof producing accurate results) as well as with a Phantom Omni hapticrendering device. Using the latter, only translational forces could beperceived by the user since the Phantom Omni only provides 3 DOFs ofsensation.

To evaluate the presented haptic rendering method in a noise freeenvironment, a virtual box (with 5 sides but no top) was constructed.The normal vectors were set perpendicular to each side, pointing intothe box. The position of the haptic rendering device was then automatedto move 2 s into the box, rotate in the positive direction for 2 s,rotate in the opposite direction for 2 s, and then move for 2 s out ofthe box. In this example, the virtual tool became constrained by thesides of the box and interacted with up to 6 contact pointssimultaneously.

Two examples of haptic interaction with streaming point data wereperformed using arbitrary polygon models as virtual tools. Theperformance of the haptic rendering method can depend on the number ofpoints in the neighborhood of the virtual tool. In some embodiments,most of the computational time can be spent on collision detection andforming movement constraints. In these embodiments, performance of thealgorithm can scale approximately linearly in the number of neighboringpoints. To improve performance in interaction with large tools, thepoint data can be down-sampled. Further, performance can be enhanced byutilizing multiple CPUs of a computing device for collision detection.Also, the algorithm can be implemented partially or completely usinginterval arithmetic, which may improve performance as well.

Example Technique for Generating and Enforcing Virtual Fixtures

FIG. 5A depicts points 510 derived from depth data and forbidden-regionfixture 514, in accordance with an example embodiment. FIG. 5A showspoints 510 including points 512 a-512 i. Forbidden-region fixture 514can define a forbidden region that includes some of points 510; e.g.,points 512 d-512 f as shown in FIG. 5A, and surrounding space where avirtual tool is prohibited from entry.

A forbidden region can be generated by selecting points, planes, regionsof space, and/or objects in a virtual environment to define theforbidden region. For each forbidden-region point p_(i) completelywithin the forbidden region, a forbidden-region radius r_(f,i) and aforbidden-region stiffness k_(s, i) can be determined. FIG. 5A shows atwo dimensional forbidden region feature defined by forbidden-regionpoints 512 d-512 f and where r_(f,1)=r_(f,2)=r_(f,3). During hapticrendering of points 510, a virtual tool will be prohibited from enteringforbidden region 514.

During haptic rendering, the forbidden regions can be considered to bedefined by forbidden-region spheres of the forbidden-region fixture,each forbidden-region sphere defined by a forbidden-region point p_(i)and related forbidden-region radius r_(f,i). Then, the virtual tool canbe constrained to only move on the surface of or away from theforbidden-region spheres.

FIG. 5B depicts virtual tool 520, in accordance with an exampleembodiment. Virtual tool 520 has a center 522, shown with a V in FIG.5B. Virtual tool 520 also has contact point(s) 524, defined as one ormore points on an exterior surface of virtual tool 520.

The definitions of virtual-tool states can be modified to account forforbidden regions. The virtual tool can be considered to be in afree-motion state with respect to one or more forbidden regions if thevirtual tool is not on the boundary or inside of any of the one or moreforbidden regions. The definition of depth can be similarly modified, sothat a modified depth MD of a virtual tool VT with respect to aforbidden region F having outward normal FN can be specified as distancebetween contact point(s) of VT and a closest portion of forbidden regionF moving in a direction of FN. In terms of the modified depth, VT is ina free-motion state with respect to forbidden region if the modifieddepth MD between VT and F is greater than 0.

The virtual tool can be considered to be in an in-contact state withrespect to one or more forbidden regions if a portion of one or more ofthe forbidden region(s) is just touching contact point(s) 524; e.g., themodified depth MD between the virtual tool and the forbidden regionequals 0. The virtual tool can be considered to be in an in-collisionstate with respect to one or more forbidden regions if a portion of oneor more of the forbidden region(s) is within contact points of thevirtual tool e.g., the modified depth MD between the virtual tool andthe forbidden region is less than 0.

When the virtual tool is in a free motion state, movement along thevector of unconstrained acceleration is allowed as long as the virtualtool does not come in contact with or collide with a forbidden region;e.g., as long as the modified depth remains greater than zero.

FIG. 5C shows an example virtual tool 530 whose movement toward desiredvirtual tool position 550 is impeded by forbidden region 540, inaccordance with an example embodiment. As shown in the example of FIG.5C, forbidden region 540 centered at point p_(i) can constrain movementof free-motion virtual tool 530 toward desired virtual tool position 550in the direction of vector u. Specifically, FIG. 5C shows that when acenter of virtual tool 530 moves to a position P₂, virtual tool 530 willbe in contact with forbidden region 540 at point p_(c). The outwardnormal to forbidden region N(p_(c)) indicates a direction of motion forvirtual tool 530 to avoid forbidden region 530.

FIG. 5D shows an example virtual tool 534 in collision with forbiddenregion 560 centered at point p_(j), in accordance with an exampleembodiment. As shown in the example of FIG. 5D, a portion of virtualtool 534 is inside of, and so is in collision with, forbidden region560. For example, forbidden region 560 can be generated based on newdepth information received from a depth-enabled camera or similardevice. In particular, point p_(c) of the contact points for virtualtool 534 is a point furthest inside forbidden region 560; e.g., a pointof virtual tool 534 closest to point p_(j) centering forbidden region560. As such, virtual tool 534 cannot travel along vector u2 towarddesired virtual tool position 570 without at least partially traversingforbidden region 560. A vector u3 from center point p_(j) to contactpoint p_(c) indicates a direction of motion for virtual tool 535 toescape forbidden region 560; e.g., virtual tool 534 can move along aweighted vector sum of u2 and u3 to escape forbidden region 560 whileprogressing toward desired virtual tool position 570.

Example Applications Involving Haptic Rendering

FIG. 6 is an example gaming scenario 600 with haptic rendering inenvironment 610, in accordance with an example embodiment. In scenario600, players 620 and 630 are playing a game of “capture the flag” ingame environment 610. FIG. 6 shows player 620 as a black circle in gameenvironment 610 and guarding flag 622, and shows player 630 as a greycircle in game environment 610 and guarding flag 632.

In capture the flag, each player tries to be the first player to capturethe opponent's flag and return the captured flag to a goal area. Thatis, during the game, player 620 attempts to capture flag 632 and returnflag 632 to goal 624 before player 630 captures flag 622 and returnsflag 622 to goal 634. In the variation of capture the flag shown inscenario 600, a player can lose if a shield level for the player goes to0% as well.

In scenario 600, both players 620 and 630 utilize haptic feedbackdevices, such as haptic gloves, body suits with haptic feedbackgenerators, and/or other haptic devices. Also, players 620 and 630 eachhave computing devices configured to use game-playing software access avirtual environment generated from depth data generated by depth-enabledcameras 650, 652, 654, 656, 658, 660, 662, and 664 (labeled as C's inFIG. 6). In other scenarios, more or fewer cameras can be used thanshown in FIG. 6. Also, in scenario 600, both players 620 and 630 havesensors configured to provide location and heading information for theplayer. For example, both players 620 and 630 could have a haptic suitwith a portable display that includes Global Positioning System (GPS)sensor(s), accelerometer(s), gaze detection sensors, and/or othersensors to both access the virtual environment, provide location andheading information, and receive haptic feedback.

Software for the game can determine a location and heading for eachplayer within environment 610 and generate a display of environment 610at the player's location as the player looks in the heading direction.FIG. 6 shows display 670 generated for player 620 with a view of box 640shown with a black star. In scenario 600, a “charging object”, or starhaving the player's color on an object within environment 610 increasesor “charges” the player's shield level as long as the player touches theobject. In contrast, if the player touches a “discharging object”, orobject having a star of the color of the opponent, the player's shieldlevel decreases or “discharges.” In some embodiments, a dischargingobject can produce a forbidden zone that discharges a shield level whilea player is within the forbidden zone. The colored star used foridentification of charging and discharging objects can be generated bythe game-playing software to permit changes in colors of player and/orchanging which objects act as charging and/or discharging objects. Also,in some embodiments, touching a discharging object can quickly or evenimmediately reduce a shield level to 0—e.g., the discharging objectcauses the player to (nearly) instantly lose the game. Depending on thescenario, such instant-discharge objects may or may not be identified tothe player by the game-playing software.

Along with identification of charging and discharging objects, thegame-playing software can generate slogans, images, etc. on objects inthe environment. For example, in scenario 600, player 620 has a headingof due south, and the game-playing software has generated the slogan“Lose any flags?” on an image of object 644 in display 670. Similarly,in scenario 600, display 680 for player 630 has slogans “Dare to pass?”and “Run!” display on generated images of objects 644. In somescenarios, the game-playing software may provide images of objectswithout additional slogans, images, etc.

Displays 670 and 680 each include game-related data for each player,including a shield level, a number of flags taken, and a number ofopponents in environment 610. For example, FIG. 6 shows display 670 forplayer 620 with game-related data that indicates that player 620: (a)has a shield level of 98% and that the shields are charging, (b) has nottaken any flags, and (c) has one opponent in environment 610. FIG. 6also shows display 680 for player 630 with game-related data thatindicates that player 630: (a) has a shield level of 100%, (b) has nottaken any flags, and (c) has one opponent in environment 610. In someembodiments, more, different, and/or less game-related data can beprovided in a display for a player.

In some embodiments, the game-playing software can generate and/ordisplay virtual objects as well as, or instead of, actual objects suchas objects 640, 642, 644, 646, and 648 of environment 610. For example,objects 648 a and 648 b, each shown using dashed lines, can be virtualobjects. As one example, a virtual object can represent a “regular”object similar to object 646 or 648 but is not physically present inenvironment 610. In other examples, a virtual object can represent atrap such as a “covered pit” that an unwary player can fall into andlose shield strength, or a treasure such as “supercharger” that canimmediately add shield strength to the player.

Many other examples of real and virtual objects are possible, includingexamples of other games that utilize haptic rendering. Some of thesegames can involve only one player and some games can involve more thantwo players. For example, a maze game with haptic feedback can involve asingle player exploring the maze or can involve two or more playersperhaps running a race through the maze.

FIG. 7 is an example haptic rendering session scenario 700 in a remoteenvironment 710, in accordance with an example embodiment. Environment710 includes a dog “Fido” 720, a depth-enabled camera 730, and ahaptically-controlled device (HCD) 732. During scenario 700, a remoteviewer interacts with Fido 720 via HCD 732. For example, HCD 732 can bea mobile robot with a robot arm configured for remote control. OtherHCDs are possible as well. In some scenarios, HCD 732 is not present,which permits interaction with a virtual environment generated by datafrom camera 730 without the ability to affect a real environment, suchas environment 710.

Display 740 can be generated to visualize a virtual environmentutilizing depth data generated by depth-enabled camera 730. FIG. 7 showsdisplay 740 with visualization portion 742 configured to display a viewof environment 710 and textual portion 744 configured to display bothinstructions, such as “Move your glove to pet Fido where the black handtouches him”, and other information such as “If you′re good to him, Fidomight take you for a walk in the woods.”

A user conducting a haptic rendering session scenario can use a hapticglove or other haptic interface device to control virtual tool 746 andtouch Fido 720 using the robot arm of HCD 732. In some scenarios, theuser can control movement of HCD 732, perhaps by certain movements ofthe haptic interface device; e.g., press a “move robot” button orotherwise signal a change of interpretation of movements of the hapticinterface device from being interpreted as commands for moving the robotarm to commands for moving HCD 732.

In scenario 700, the move robot button can be pressed to move HCD 732along path 734 and so walking Fido 720 based on movements of the hapticinterface device (e.g., move haptic interface device left or right tocorrespondingly move HCD 732 left or right along path 734, rotate hapticinterface device to better pet Fido). When the move robot button is notpressed, movements of the haptic interface device control the robot arm(e.g., for a haptic glove acting as the haptic interface device, Fido720 can be petted or scratched based on finger movements of the hapticglove).

In other scenarios, non-haptic interface devices can be used to controlHCD 732; e.g., the haptic interface device controls the robot arm and ajoystick or keyboard can be used to move the mobile robot. In stillother scenarios, camera 730 can be mounted on HCD 732 to provideadditional information about environment 710, including informationabout Fido 720.

FIG. 8 is an example collaborative haptic rendering session scenario 800for a remote environment 810, in accordance with an example embodiment.Environment 810 includes pumping station 812 that, during scenario 800,has failed. Due to the failure of pumping station 812, waste 814 hasescaped, as shown by FIG. 8 as a black cloud in environment 810.

During scenario 810, four robots 820, 830, 840, and 850 have beendeployed to fix pumping station 810 and investigate environment 810 tobegin cleaning waste 814. Each of robots 820, 830, 840, and 850 has adepth-enabled camera facing forward and can be driven by a remoteoperator using a display, such as display 312 discussed above in thecontext of FIG. 3A, and one or more haptic interface devices, such ashaptic interface device(s) 316 discussed above in the context of FIG.3A.

In scenario 800, each robot operator is in a different physicallocation. Each location is equipped with one or more haptic interfacedevices, one or more displays, and perhaps other interface devices, suchas keyboards, keypads, touch screens, loudspeakers, microphones, and/orother interfaces. In other scenarios, some or all of the robot operatorscan be in the same remote location. In still other scenarios, a singlerobot can be used with a single robot operator. In even other scenarios,one or more of the robots can be controlled by multiple robot operatorse.g., both a driver and a ladder operator can control a robotic fireengine. In yet other scenarios, one or more of the robot operators canbe local; e.g., at environment 810, perhaps riding on or within a robot.

FIG. 8 shows the displays generated the depth-enabled cameras for therobot operators, as display 860 for robot 820 headed along heading 826,display 870 for robot 830 headed along heading 836, display 880 forrobot 840 headed along heading 846, and display 890 for robot 850 headedalong heading 856. Robot 820 has two robot hands configured as hapticinterface devices (HIDs) 822, 824, robot 840 has two robot hands 842,844 configured as haptic interface devices, and robot 850 has robot hand852 and probe 854 configured as haptic interface devices. Each robotdisplay shown in FIG. 8 includes a visualization portion and a textualportion, similar to display 1040 shown in FIG. 10. For example, display860 includes visualization portion 862 and textual portion 864. In FIG.8, virtual tools representing robot hands are shown using block diagramsof hands with two fingers and virtual tools representing probes areshown using slender triangles; e.g., pennant-shaped triangles.

In scenario 800, robot 830 is in communication with the other robots820, 840, and 850, and is configured to a “communication coordinator” tosend and receive text, voice, and perhaps other types of messages fromthe other robot operators. For example, robot 830 can be controlled by asupervisor observing environment 810 and coordinating the efforts ofrobots 820, 840, and 850.

In scenario 800, a far end of each robot arm of robot 820 is configuredto be a virtual tool for the operator of robot 820. FIG. 8 shows display860 for robot 820 including a location of a left virtual tool 866,corresponding to a left robot hand on a left arm of robot 820, and aright virtual tool 868 corresponding to a right robot hand on a rightarm of robot 820. In FIG. 8, right virtual tool 868 is position to movea handle of a door to pumping station 812.

The operator of robot 820 can use a non-haptic interface device, such asa keyboard or keypad to enter text that appears in textual portion 864.FIG. 8 shows that the text “Joe, I'm trying to open the R door.” Is intextual portion 864, preceded by an identifier “Chris” of a person whoentered in the text; e.g., Chris is a first name of the operator ofrobot 820. In scenario 820, text entered into a textual portion 864 ofdisplay 860, textual portion 884 of display 880, or textual portion 894of display 890 is displayed both in the entering textual portion plus intextual portion 874, as display 870 and textual portion 874 areassociated with the communication coordinator. In some embodiments,robot operators can send a message to one or more other operatorsdirectly without sending a copy of the message to the communicationcoordinator.

In other embodiments, the operator of robot 820 can receive touchfeedback while attempting to open the door of pumping station 812. Forexample, as the operator of robot 820 uses robot hand 822 to open thedoor of pumping station 812, haptic rendering can provide feedback toindicate that the door is or is not resisting the efforts to open thedoor. Further, in embodiments where robot hand 822 is equipped withfinger-like appendages, the operator of robot 820 can use a haptic gloveor other haptic interface device to move the appendages to grab the doorhandle, and pull down or push up with an amount of effort based on,e.g., proportional to, an amount of effort exerted by the operator inmoving the haptic interface device.

Robot 850 is equipped with probe 854. A probe can be equipped with oneor more sensors for chemical, biological, and/or radiation testing. FIG.8 shows robot 850 with probe 854 in waste 814.

Each of displays 880 (for robot 840) and 890 (for robot 850) shows avirtual tool associated with either a robot hand or a probe. Forexample, for robot 840, robot hand 842 is associated with virtual tool886 and robot hand 844 is associated with virtual tool 888. In scenario800, both robot hands 842 and 844 include probes.

Robot 840 is engaged in measuring waste densities with robot hands 842and 844. FIG. 8 shows textual portion 884 of display 880 displayingvarious types of information, including location information for eachprobe and waste density values detected by each probe. In scenario 800,a waste density of “IP” indicates a probe is in the process ofdetermining a waste density, such as shown in FIG. 8 for probe 844.

Both haptic and non-haptic commands can be used to control robots.Textual portion 894 of display 890 shows commands being communicatedfrom an operator to robot 850. FIG. 8 shows a “Cmd:” prompt forcommands, such as “set left to rad”, which directs robot 850 to set aprobe 854 to act as a radiation sensor. Robot 850 responds with aconfirmatory message from “Probe” of “Set for rad” to indicate thatprobe 854 is ready for radiation testing. In scenario 800, the operatorof robot 850 sends another command “test probe” to robot 850. Inresponse, probe 854 performs a radiation test and determines a value of“+1.3” for the radiation test. In scenario 800, the operator of robot850 can move robot hand 852 and probe 854 using respective hapticinterface devices.

FIG. 9 depicts scenario 900 where construction is ongoing in environment910, in accordance with an example embodiment. Environment 910 can be avirtual environment or an actual environment. In some embodiments,environment 910 can represent an extra-terrestrial environment; e.g., inspace or on a planet other than Earth. In these embodiments, values ofvarious parameters, such as the speed of gravity (if any), planetarytilt, and relative location of other celestial bodies, can change fromEarth-normal values.

In scenario 900, a solar power array is being constructed from at leasttwo depicted sub-arrays: sub-array T1-182, shown on the left side ofenvironment 910, and sub-array T2-182, shown on the right side ofenvironment 910. To construct the solar power array, sub-array T1-182 isbeing manipulated by tool 920 and sub-array T2-182 is being manipulatedby tool 922. In other scenarios, more or fewer tools can be utilized inenvironment 910.

In scenario 900, display 940 is used to monitor and control a hapticinteraction session with tool 920 and display 950 is used to monitor andcontrol tool a haptic interaction session with tool 922. Display 940includes environmental-display region 942 and textual-display region944. Environmental-display region 942 includes a display of sub-arraysT1-182 and T2-182, virtual tool 946 representing tool 920, and twocontrol buttons. The “Bye” control button ends the haptic interactionsession and the “Change Tool” button enables changing display 940 tomonitor/control to a different virtual tool and/or to change operationof tool 920; e.g., changing use of a robot hand as part of tool 920 asshown in FIG. 9. More, fewer and/or different control buttons can beutilized in other scenarios. Textual-display region 944 can provide textand perhaps other types of information (e.g., images, sounds, sensorinformation, communication sessions, etc.) for use as part of display940.

Virtual tool 946, and subsequently tool 920, can be controlled by use ofa 6 DOF (or other) haptic interface to enable control over location androtation of virtual tool 946 and to receive force feedback related torelated to the virtual tool using the herein-described techniques.

In other scenarios, display 940 can be used to monitor and controlmultiple haptic interaction sessions: for example, a robot arm can havethree virtual tools corresponding to a shoulder, an elbow, and a wrist,with each virtual tool having up to 6 DOF. Then, if display 940 wereused to control the robot arm, display 940 can control hapticinteraction sessions for each of the virtual tools. As another example,a tool can have multiple controllable components; e.g., robot arms,robot hands, an engine/propulsion system, etc., where some or all of thecontrollable components can be directed using haptic interactionsession(s). Then, if display 940 were used to monitor and control such atool, display 940 can control one or more haptic interaction session(s)with the tool. Many other examples are possible as well.

Display 950 is similar to display 940. Display 950 includesenvironmental-display region 952 and textual-display region 954.Environmental-display region 952 includes a display of sub-arrays T1-182and T2-182, virtual tool 956 representing tool 922, and “Bye” and“Change Tool” control buttons discussed above in the context of display940. Textual-display region 954 can provide text and perhaps other typesof information for use as part of display 950.

In scenario 900, tools 920 and 922, controlled via respective displays940 and 950 and corresponding haptic interaction sessions, construct thesolar power array using sub-arrays T1-182 and T2-182. Scenario 900 canthen end.

Underwater Haptic Rendering Applications

Virtual tools, haptic interaction sessions, and the herein-describedtechniques can be applied to underwater applications. For example, aunderwater depth-enabled camera, similar in spirit to the Microsoft XboxKinect, but suitable for underwater conditions can be used for locatingobjects underwater and providing data for 6 DOF haptic feedback to ahuman operator of the robot. The haptic feedback can provide a humanoperator with a sense of touch for objects seen by the camera.

This system can: (a) permit the operator to ‘feel’ objects, such asunderwater ordnance during remediation or objects being salvaged,through a haptic 6 DOF interface, based upon the camera system image anddynamic haptic rendering, (b) enable the operator to guide the robotend-effectors with the target during removal, via tele-operation, and(c) establish virtual ‘force fields’ around protected zones of objects(such as locations that might result in explosion of the ordnance). Ifthe tele-operator tries to move an end-effector too close to a protectedzone, he/she will feel resistance as the haptic interface “pushes back”.Imposition of protected zones, perhaps via virtual features, can preventrobot end effector contact with undesirable locations, either in theenvironment or on the ordnance. The end effector of a robot can be adevice at an end of a robot arm or other appendage; e.g., a robotic handat the end of robot arm. Combined with a tele-operated robotic device,this will allow human directed robotic capture and removal of objectsunder fresh or salt water.

FIG. 10 shows underwater robot 1000, in accordance with an exampleembodiment. Robot 1000 is configured with two screw drives 1002, 1004,eight robotic arms 1010, 1011, 1012, 1013, 1014, 1015, 1016, 1017, lightsources 1020, 1024, and camera 1022.

Robot 1000 can be a bottom-crawling vehicle with a screw-drivepropulsion system. Robot 1000 can move over an underwater object, suchas ordnance, and the tele-operator uses robot arms 1010-1017 to grab theunderwater object. Light sources 1020, 1024 and camera 1022 can operatetogether as a camera system.

As shown in FIG. 10, camera system 1020, 1022, 1024 can be integratedinto a chassis of robot 1000 and image the region within service bay1030. Camera system 1020, 1022, 1024 can generate a three-dimensionalimage of object(s) within service bay 1030, such as object 1032. Theimages generated by camera system 1020, 1022, 1024 can be provided to aprocessor at the surface to haptically rendering images within servicebay 1030. A human tele-operator interacts with robot 1000, arms1010-1017, and objects such as object 1032, using one or more hapticinterfaces and perhaps display(s). The tele-operator can control arms1010-1017 to move object 1032, pick up object 1032, attach a sling andcable to object 1032 for future retrieval, and/or some other action suchas mark the object for better identification.

Robot 1000 can use a system of screw drives 1002, 1004 for propulsion.Using screw drives maintains negative buoyancy throughout a mission andthus robot 1000 need not manage buoyancy. The use of screw drives, incontrast to thrusters, lowers the risk of stirring up silt or sand,which can deteriorate visibility. Additionally, silt poses a challengeto traditional track-driven vehicles not faced by screw drives.

FIG. 10 shows that screw drives 1002, 1004 are mounted on each side ofrobot 1000. Each screw drive 1002, 1004 can be driven at varying speeds.Robot 1000 can move sideways and thrust may be vectored so thatnon-holonomic constraints do not limit operation. Thrust vectoringallows for increased maneuverability and simplifies the role of thetele-operator, allowing for natural modes of operation and quickadjustments to the platform's location on the sea, river, or lake bed.Screw drives 1002, 1004 can also be articulated, allowing for a range ofmaneuvers and for convenient movements such as vertical translation ofthe chassis and service bay 1032 of robot 1000. In some scenarios, screwdrives 1002, 1004 can dig into mud, getting the robot 1000 deeper intoareas that may have buried ordnance.

Robotic manipulation and removal of munitions requires a stable platformthat can remain fixed in an inertial frame with respect to the object.To carefully and firmly grip an object identified for removal, thesystem can use a number of robotic manipulators designed for the task,such as arms 1010-1017.

Arms 1010-1017 can have a series of linkages and the necessary actuationto achieve motion derivatives that accomplish desired gripping maneuversto handle underwater objects. In some embodiments, such as shown in FIG.10, each of arms 1010-1017 is identical and is designed for planarmotion. When gripping an object, the trajectory of each of arms1010-1017 is designed to achieve optimal grasping with respect to thegoals and constraints defined by a human operator.

In some embodiments, some of arms 1010-1017 can be controlled using alow number of degrees of freedom. Even with low degree-of-freedom arms,intricate shapes can still be achieved with the coordinated manipulatorsystem to allow for a high level of dexterity using simple components.As shown in FIG. 10, arms 1010-1017 can be mounted along either side ofthe service bay 1030, allowing them to oppose on another so that simplemotions can result in stable gripping of munitions. The use of eightarms allows for a lower contact pressure on an object from eachcontacting arm. Also, a subset of arms 1010-1017 can be used, such aswhen a forbidden-region fixture on object 1032 prohibits one or morearms from gripping object 1032 at a particular location. In someembodiments, robot 1000 and arms 1010-1017 can operate in several modesof operation and redundancy, allowing an operator to carry out taskseven when a manipulator suffers damage.

The camera system can use a ‘range gated’ approach to allow both visible(blue green) and NIR images to be captured in combined stream of visible(blue green) and NIR images. Light source 1020 can be configured withblue-green filters and/or blue-green light-emitting diodes. Theblue-green filters are configured to pass through light in a blue-greenfrequency range of 450-480 nm, while the blue-green diodes areconfigured to emit light in the blue-green frequency range.

Light source 1024 can include a NIR laser and a diffraction grating. TheNIR laser and diffraction grating project a pseudo-random of dots ofvarying intensities. Camera 1022 can measure depth from the distortionbetween the projected and observed dot patterns. In some embodiments,the pulse duration for the NIR laser can be shorter than the time totravel to the target to reduce back scatter.

Camera system 1022 can be configured to capture the visible (blue green)and NIR images alternatively; that is illumination and camera frames aresynchronized to capture the visible (blue green) and NIR imagesalternatively. That is, light sources 1020, 1024 can be alternativelyactivated at frame speed of camera 1022 to capture visible and NIRimages in a single image stream. Dual-wavelength camera system 1022 canallow robot 1000 to obtain 2D images with depth measurements in lowlight conditions using a single camera. In some embodiments, camerasystem 1022 can be configured as an endoscope. Camera 1022 can includeone or more adapters to combine the projected mask pattern from thelight source 1024 with visible light source 1020, and recover the depthinformation from the reflected light with a wavelength-specific beamsplitter.

In some embodiments, camera 1022 can be a modified 0° or 30° 10 mmendoscope with light sources and detectors separated by fixed distances.In particular embodiments, camera 1022 can output frames with VGAresolution (720×900 pixels) at 30 frames per second, in RGB+Depth(RGB+D) format.

To calibrate the visible and depth images, planar calibration of camerasystem 1020, 1022, 1024 can first be performed on a planar surface witha checkerboard pattern. With camera system 1020, 1022, 1024 configuredfor close depth detection, the checkerboard can be produced withphotolithographic techniques for both scale and accuracy. After planartesting is complete, optical testing of camera system 1020, 1022, 1024can be performed on an irregular non-planar surface to determine a depthresolution, and any effect of geometric distortion on the opticaldistortion of camera system 1020, 1022, 1024. If distortion is detected,a distortion-correction technique that automatically calculatescorrection parameters without precise knowledge of horizontal andvertical orientation can be used to correct the detected distortion.

Camera system 1020, 1022, 1024 is discussed in more detail below in thecontext of at least FIGS. 17 through 29.

The above-mentioned haptic rendering process can use camera system 1020,1022, 1024 to generate depth data, depth images, and haptic feedback.The haptic feedback can give a tele-operator a “feel” for underwaterobjects that are within the field of the underwater video+depth cameras,such as underwater ordnance during remediation. The tele-operator'sconsole (to transmit operator control actions as well as hapticfeedback) can use two haptic interface devices such as discussed abovein the context of FIG. 3A. As the tele-operator moves the robot throughthe haptic interface devices, he/she can feel a “force field” ofincreasing impedance, as proximity of the tool tip to the virtualfixture boundary decreases.

Virtual fixtures can be established around the portion of protectedstructure; that is, structures not to be touched during a remediationprocedure. Force-feedback virtual fixtures designed can improve theeconomy, speed and accuracy of user motions in the tele-operatedenvironment. In particular, forbidden-region fixtures (FRFs) driven byhaptic rendering information obtained from depth-enabled camera(s) canbe used in two feedback control paths in this co-robotic system: by thetele-operator, and by a position control system for robot 1000.

Virtual fixtures around critical parts of a target (e.g., locations thatwould trigger explosion of the ordnance) can be designated by operatorinput or through image recognition. For operator designation of avirtual fixture, the tele-operator will specify the boundaries ofvirtual fixture either using a haptic interface device, or by usingmouse, touch screen, or other input on a video display. At any timeduring operation of robot 1000, a tele-operator of robot 1000 canre-define virtual fixtures by drawing on a real time image, or byrepeating the virtual fixture designation process. For automaticrecognition, an image recognition capability could be used to specify‘no touch’ zone(s) based on objects detected from images captured bycamera system 1020, 1022, 1024.

Effectors at ends of robot arms 1010-1017 can be tracked in real time.Using the haptic rendering algorithms described herein, haptic feedbackcan be provided to the tele-operator, such as pushing back if aneffector gets too close to a protected location; i.e., near or within aforbidden region defined by a virtual fixture, such as aforbidden-region fixture. Additionally or instead, if an effector getstoo close to a protected location, robot control actions can be modifiedto lock out certain motions. For example, suppose a forbidden region hadbeen defined that was directly astern from robot 1000. Then, if thetele-operator of robot 1000 attempted movement of one or more of arms1010-1017 backwards close to or within the forbidden region, hapticfeedback, such as resistance, can be used to inform the tele-operatorabout the forbidden region. Additionally or instead, backward movementsof robot 1000 can be inhibited while the forbidden region remainsdirectly astern of robot 1000. In some embodiments, both providinghaptic information and dynamic robot responses can be modified; such asboth providing resistance to the tele-operator and slowing down motionof robot 1000 that is near or within a boundary of a virtual fixture.Many other examples are possible as well.

An Example Computing Network

FIG. 11A depicts a network 1100 in accordance with an exampleembodiment. In FIG. 11A, servers 1108 and 1110 are configured tocommunicate, via a network 1106, with client devices 1104 a, 1104 b, and1104 c. As shown in FIG. 11A, client devices can include a personalcomputer 1104 a, a laptop computer 1104 b, and a smart-phone 1104 c.More generally, client devices 1104 a-1104 c (or any additional clientdevices) can be any sort of computing device, such as a workstation,network terminal, desktop computer, laptop computer, wirelesscommunication device (e.g., a cell phone or smart phone), and so on.

The network 1106 can correspond to a local area network, a wide areanetwork, a corporate intranet, the public Internet, combinationsthereof, or any other type of network(s) configured to providecommunication between networked computing devices. Servers 1108 and 1110can share content and/or provide content to client devices 1104 a-1104c. As shown in FIG. 11A, servers 1108 and 1110 are not physically at thesame location. Alternatively, servers 1108 and 1110 can be co-located,and/or can be accessible via a network separate from network 1106.Although FIG. 11A shows three client devices and two servers, network1106 can service more or fewer than three client devices and/or more orfewer than two servers.

An Example Computing Device

FIG. 11B is a block diagram of an example computing device 1120including user interface module 1121, network-communication interfacemodule 1122, one or more processors 1123, and data storage 1124, inaccordance with embodiments of the invention.

In particular, computing device 1120 shown in FIG. 11A can be configuredto perform one or more functions of client devices 1104 a-1104 c,networks 310, 1106, 1720, servers 1108, 1110, computing devices 320 a,320 b, 1724, robot 1000, 1784, software 1310, task-management software,master console 1710, camera 1780, 1800, depth sensor 1782, sensor system1860, 2600, 2630, system 2200, 2300, metal detector 2420, 2460, and/ormetal detector system 2500, 2520, 2540, and/or to implement part or allof architecture 1300, pipeline 1400, remote virtual environment 1704,and/or scenario 2800. Computing device 1120 can include a user interfacemodule 1121, a network-communication interface module 1122, one or moreprocessors 1123, and data storage 1124, all of which may be linkedtogether via a system bus, network, or other connection mechanism 1125.

Computing device 1120 can be a desktop computer, laptop or notebookcomputer, personal data assistant (PDA), mobile phone, embeddedprocessor, or any similar device that is equipped with at least oneprocessing unit capable of executing machine-language instructions thatimplement at least part of the herein-described techniques and methods,including but not limited to method 1200 described in more detail belowwith respect to FIG. 12.

User interface 1121 can receive input and/or provide output, perhaps toa user. User interface 1121 can be configured to send and/or receivedata to and/or from user input from input device(s), such as a keyboard,a keypad, a touch screen, a computer mouse, a track ball, a joystick,and/or other similar devices configured to receive user input from auser of the computing device 1120. User interface 1121 can be configuredto provide output to output display devices, such as, one or morecathode ray tubes (CRTs), liquid crystal displays (LCDs), light emittingdiodes (LEDs), displays using digital light processing (DLP) technology,printers, light bulbs, and/or other similar devices capable ofdisplaying graphical, textual, and/or numerical information to a user ofcomputing device 1120. User interface module 1121 can also be configuredto generate audible output(s), such as a speaker, speaker jack, audiooutput port, audio output device, earphones, and/or other similardevices configured to convey sound and/or audible information to a userof computing device 1120. As shown in FIG. 11B, user interface module1121 can be configured with haptic interface 1121 a that can receiveinputs related to a virtual tool and/or a haptic interface point (HIP),a remote device configured to be controlled by haptic interface 1121 a,and/or other inputs, and provide haptic outputs such as tactilefeedback, vibrations, forces, motions, and/or other touch-relatedoutputs.

Network-communication interface module 1122 can be configured to sendand receive data over wireless interfaces 1127 and/or wired interfaces1128 via a network, such as network 1106. Wired interface(s) 1128, ifpresent, can comprise a wire, cable, fiber-optic link and/or similarphysical connection to a data network, such as a wide area network(WAN), a local area network (LAN), one or more public data networks,such as the Internet, one or more private data networks, or anycombination of such networks. Wireless interface(s) 1127 if present, canutilize an air interface, such as a ZigBee, Wi-Fi, and/or WiMAXinterface to a data network, such as a WAN, a LAN, one or more publicdata networks (e.g., the Internet), one or more private data networks,or any combination of public and private data networks.

In some embodiments, network-communication interface module 1122 can beconfigured to provide reliable, secured, and/or authenticatedcommunications. For each communication described herein, information forensuring reliable communications (i.e., guaranteed message delivery) canbe provided, perhaps as part of a message header and/or footer (e.g.,packet/message sequencing information, encapsulation header(s) and/orfooter(s), size/time information, and transmission verificationinformation such as CRC and/or parity check values). Communications canbe made secure (e.g., be encoded or encrypted) and/or decrypted/decodedusing one or more cryptographic protocols and/or algorithms, such as,but not limited to, DES, AES, RSA, Diffie-Hellman, and/or DSA. Othercryptographic protocols and/or algorithms can be used as well as or inaddition to those listed herein to secure (and then decrypt/decode)communications.

Processor(s) 1123 can include one or more central processing units,computer processors, mobile processors, digital signal processors(DSPs), GPUs, microprocessors, computer chips, and/or other processingunits configured to execute machine-language instructions and processdata. Processor(s) 1123 can be configured to execute computer-readableprogram instructions 1126 that are contained in data storage 1124 and/orother instructions as described herein.

Data storage 1124 can include one or more physical and/or non-transitorystorage devices, such as read-only memory (ROM), random access memory(RAM), removable-disk-drive memory, hard-disk memory, magnetic-tapememory, flash memory, and/or other storage devices. Data storage 1124can include one or more physical and/or non-transitory storage deviceswith at least enough combined storage capacity to containcomputer-readable program instructions 1126 and any associated/relateddata structures.

Computer-readable program instructions 1126 and any data structurescontained in data storage 1126 include computer-readable programinstructions executable by processor(s) 1123 and any storage required,respectively, to perform at least part of herein-described methods,including but not limited to method 1200 described below with respect toFIG. 12, method 1600 described below with respect to FIG. 16, and/ormethod 2200 described below with respect to FIG. 22. Computer-readableprogram instructions 1126 can include instructions that when executed byprocessor(s) 1123 to perform the herein-described functionality ofsoftware, such as, but not limited, to software 1310 and/ortask-management software.

An Example Method for Six Degree of Freedom Haptic Rendering

FIG. 12 is a flowchart depicting example functional blocks of an examplemethod 1200. Method 1200 begins at block 1210, where a computing devicecan receive first depth data about an environment, such as discussedabove in detail in the context of at least Table 1 and FIGS. 3A and 3B.

At block 1220, the computing device can generate a first plurality ofpoints from the first depth data, such as discussed above in detail inthe context of at least Table 1 and FIGS. 3A-4C.

In some embodiments, such as discussed above in the context of at leastTable 1 and FIGS. 3A-4C, the first depth data can include data about aninput plurality of points in the environment. Then, determining thefirst plurality of points can include: generating a filtered pluralityof points by filtering the input plurality of points using a bilateralfilter; and determining a normal vector for each point in the filteredplurality of points.

In particular of these embodiments, the computing device can include agraphics processing unit (GPU). Then, determining the normal vector foreach point in the filtered plurality of points can include determiningthe normal vector for each point in the filtered plurality of pointsusing the GPU.

At block 1230, the computing device can determine a virtual tool, suchas discussed above in detail in the context of at least Table 1 andFIGS. 3A, 5A-5D, 7, 8, and 9. The virtual tool can be specified in termsof a translation component for the virtual tool and a rotation componentfor the virtual tool. In some embodiments, the translation component canbe specified in terms of up to three degrees of freedom, the rotationcomponent can be specified in terms of up to three degrees of freedomdistinct from the three degrees of freedom used for the translationcomponent, and thus the virtual tool can be specified in terms of up tosix degrees of freedom.

In other embodiments, the translation component can relates to aposition of the virtual tool and the rotation component can relate to arotation of the virtual tool about at least one axis. In still otherembodiments, determining the virtual tool can include representing thevirtual tool using a plurality of voxels, where each voxel has a voxelsize in each of three dimensions, and where the voxel size for each ofthe three dimensions is a same size.

At block 1240, the computing device can determine a first force vectorbetween the virtual tool and the first plurality of points, such asdiscussed above in the context of at least Table 1 and FIGS. 3A, 5A-5D,6, 7, and 10.

In some embodiments, determining the first force vector can includedetermining a bounding box for the virtual tool based on a projection ofthe virtual tool onto the first depth image; and determining neighboringpoints of the first plurality of points, wherein each neighboring pointis within the bounding box. In particular of these embodiments, method1200 can also include determining whether there are zero neighboringpoints or more than zero neighboring points; and in response todetermining that there are zero neighboring points: determining that thevirtual tool is in a free-motion state, and moving the virtual tool in adirection based on the translation component and by a distance boundedby a same voxel size.

In other particular of these embodiments, method 1200 can also include:in response to determining that there are more than zero neighboringpoints: determining a depth for each neighboring point with respect tothe virtual tool; determining in-contact neighboring points from theneighboring points, where each in-contact neighboring point has acorresponding depth of zero; determining in-collision neighboring pointsfrom the neighboring points, where each in-collision neighboring pointhas a corresponding depth less than zero; in response to determiningthat no in-collision neighboring points are determined and todetermining at least one in-contact neighboring point is determined:determining a first constrained acceleration for the virtual tool basedon the at least one in-contact neighboring point, and moving the virtualtool in a direction based on the first constrained acceleration.

In still other particular of these embodiments, method 1200 can alsoinclude: in response to determining that at least one in-collisionneighboring point is determined: determining a second constrainedacceleration for the virtual tool based on the at least one in-collisionneighboring point; and moving the virtual tool in a direction based onthe second constrained acceleration.

At block 1250, the computing device can send a first indication ofhaptic feedback based on the first force vector, such as discussed abovein the context of at least Table 1 and FIGS. 3A, 5A-5D, 6, 7, and 10.

In some embodiments, such as discussed above in the context of at leastTable 1, method 1200 can include: receiving second depth data about theenvironment at the computing device, where the second depth data candiffer from the first depth data; generating a second plurality ofpoints from the second depth data using the computing device;determining a second force vector between the virtual tool and thesecond plurality of points using the computing device; and sending, fromthe computing device, a second indication of haptic feedback based onthe second force vector.

In other embodiments, such as discussed above in the context of at leastFIG. 3A, the computing device can be configured to communicate with acontrollable mechanism. Then, sending the indication of haptic feedbackbased on the force vector from the computing device can include sendingthe indication of haptic feedback based on the force vector from thecomputing device to the controllable mechanism. In these embodiments,method 1200 can also include providing haptic feedback based on theindication of haptic feedback using the controllable mechanism. Inparticular of these embodiments, the controllable mechanism can includea manipulator configured for surgical operation.

In still other embodiments, such as discussed above in the context of atleast FIG. 3A, the first depth data can include first image data andcorresponding first depth values. In these embodiments, method 1200 canalso include generating a virtual environment based on the first imagedata and corresponding depth values. In particular of these embodiments,such as discussed above in the context of at least Table 1 and FIGS.4A-4C, the virtual environment can include a first object. In theseembodiments, method 1200 can also include determining a contact betweenthe first object and the virtual tool.

In even other embodiments, method 1200 can also include defining aforbidden region within the environment. Then, determining the firstforce vector between the virtual tool and the first plurality of pointscan include inhibiting the virtual tool from moving within the forbiddenregion. In yet even other embodiments, method 1200 can also includecontrolling a motion of an object or character within a virtualenvironment, wherein the virtual environment comprises at least onevirtual object and at least one real surface.

Virtual Fixtures for Human/Autonomous Manipulation Tasks

In haptic interaction, an operator can use a physical haptic device tointeract with objects in a virtual environment. This virtual environmentcan consist of representations of physical objects, as captured bysensor data, virtual objects that are entirely computer simulated, orcombinations of representations of physical objects and virtual objects.A Haptic Interface Point (or HIP) can be thought of as the position of a3-DOF end effector in virtual space that matches the position of thehaptic device. For 6-DOF end effectors, a virtual tool can represent theend effector. When the user moves the haptic device, the HIP or virtualtool can move accordingly in the virtual environment.

Depth data can be specified as a point cloud can be thought of as anunstructured set of points representing the boundary of physicalobjects. Depth data can be obtained using stereo cameras, laser scannersor depth cameras in which case the data will be sorted in the imageplane. A depth image can augment the depth data with color information.Haptic rendering as described herein can utilize streaming point clouddata represented in depth images for 3 DOF and 6 DOF applications.

Complex telerobotic manipulation tasks require high precision andrepetitive actions for which automated systems are highly skilled.However, it also requires critical judgment and decision-making, whichis best done by a human operator. Thus, herein is described a combinedmanipulator system (human+automation) that affords the operator a greatamount of freedom but also enhances performance via computer-generatedauditory-, visual- and haptic-feedback. The resulting technology willimprove the outcome by preventing mistakes and increasing efficiency inmanipulation tasks. Rather than using the feedback to solve the task (byforcing the human operator along a pre-defined trajectory) we use thefeedback to communicate “intent” of the computer system duringsequential operations.

The feedback can be related to “virtual fixtures” added to the virtualenvironment provided to the operator utilizing haptic feedback tooperate effector(s) in a remote environment, such as an operatorperforming tasks such as remote search and rescue, sensor placement,ordnance placement/removal, remote movement of materiel, and remoteconstruction. The remote environment can be hazardous; e.g., a cleanupsite, a search and rescue location, an area with extreme weatherconditions, an underwater environment, an outer space environment.Underwater applications of this technology include human-guidedsurveillance, underwater equipment maintenance and repair, environmentalcleanup as well as tasks in the oil and gas industries. In the latter,commercial interest is high, given that expensive equipment ismanipulated in time-critical environments, where mistakes can be bothfatal and disastrous to the environment.

A virtual fixture can be defined as an overlay of abstract sensoryinformation on a workspace in order to improve the telepresence of aremotely manipulated task. Virtual fixtures can be arbitrary in bothappearance and feedback. In some scenarios, virtual fixtures can act asone or more virtual rulers guiding a remote manipulator along apre-defined path. The operator can be notified of deviations from thepath defined by the virtual fixture using audio, visual or hapticfeedback in particular. Haptic feedback (force feedback applied to theoperator control stick) can be used to guide the operator (and hence theremote end effector) back to the desired path.

Virtual fixtures can also include forbidden-region fixtures and guidancefixtures. As mentioned above, a forbidden-region fixture can define anarea represented by the point data where the virtual tool is notpermitted to enter; i.e., the forbidden region. A guidance fixture candefine an area represented by the point data where the virtual tool ispermitted, and perhaps encouraged, to enter.

Virtual fixtures, such as but not limited to forbidden-region fixturesand guidance fixtures, can be specified based on hard constraints thatprevent end effector violation of constraints, or soft constraints whereundesired motions are resisted but not prevented, using an associatedpenetration-based penalty cost. An operator can be notified aboutvirtual fixtures using any combination of visual-, auditory-, and hapticfeedback. For instance, the haptic feedback can take the form ofrepelling force feedback, vibrotactile feedback, or rendered as aviscous resistance.

Forbidden-region fixtures and guidance fixtures can be part of a set ofstandardized virtual fixtures with well-defined visual-, auditory-, andhaptic properties. This standard set of virtual fixtures can be used tospecify complex manipulation tasks in terms of common manipulationsubtasks. Example subtasks include:

-   -   Grasping—This virtual fixture will be used to aid the operator        in positioning the remote manipulator for a successful grasp of        an object of interest.    -   Rotational manipulation and translational movements—This        guidance virtual fixture will be used for valve turning and is        important to make sure that no equipment is damaged during        operation.    -   Insertion/Removal—Many operations requires coupling/decoupling        of connectors or subsea equipment, so called hot stabbing, and        it is therefore important to aid the operator in for instance        linear insertions/removals.    -   Placement—This virtual fixture will aid in operations where        equipment needs to be positioned accurately with respect to the        environment.

The virtual fixtures can be defined with respect to absolute location(s)in task space and implicitly defined by data from sensors in the remoteenvironment. The virtual fixture can adapt to moving targets and thatexternal disturbances automatically will be rejected. For example, anunderwater current can move, and thus disturb, an underwatermanipulator. In another example, an external disturbance in a surgicalcontext can occur when a patient moves (or is moved) during a surgicalprocedure. Many other examples are possible as well.

Haptic feedback can be generated based on data from non-contact sensorssuch as depth images including data from radar or sonar sensors. Usingnon-contact sensors, haptic feedback can be generated before contact hasbeen made using 3D images of objects. This can be done before physicalcontact with obstacles thus avoiding loop delays in operation. Inaddition, virtual fixtures can be defined around objects depicted in thedepth images.

By sensing range and effectively geometries near the manipulator, aguidance fixture can guide the operator to a specific location or regionand/or as a forbidden region fixture to block the operator from aspecific location or region. Virtual fixtures can be implemented toaccount for in-motion objects in the environment. Complex tasks,particularly sequential tasks, can benefit from multiple, adaptivevirtual fixtures. The virtual fixtures can change as tasks in a seriesof sequential tasks are completed—for example, a virtual fixture canstart as a guidance fixture for guiding operator to a region to performa task and then, once the task is complete, the virtual fixture canchange to a forbidden-region fixture to keep the operator from theregion to avoid undoing previously-completed work. The herein-describedsystem is capable of rendering multiple virtual fixtures and switchingbetween them in an intuitive, fluid fashion in real time, in response tothe environment and performance needs.

Virtual fixtures can aid the human operator in complex manipulationtasks. These tasks can be planned and carried out using automated andsequential transitioning between virtual fixtures. To perform part orall of the tasks, the operator can use a manipulation system with hapticrendering capabilities. During execution of the tasks, a level ofautomation can change from manual operation to fully autonomousoperation, perhaps with one or more intervening levels ofsemi-autonomous operations possible. The human operator can adjust thelevel of automation and also intervene as necessary during autonomousoperation. For example, during human-only or mostly-human operation, aflexible guidance fixture can be used that allows for small deviationsfrom a guided path, where the deviations can be subject to guiding butnon-constrained feedback, so to provide the operator increasedflexibility and control over the situation.

While the operator directs the system, the operator can also respond tosystem performance and conditions. Auditory, visual, and haptic virtualfixtures can communicate intent of the system to the operator. Forexample, virtual fixtures can be used to describe basic tasks related tomanipulation such as positioning, grasp and screwing/unscrewing tasks. Aset of virtual fixtures can be added to the environment to avoidmistakes, such as a forbidden-region fixture, a guidance fixture, astandoff fixture, which is a virtual fixture for maintaining a fixedsafety distance between the manipulator and an arbitrary object, and/ora velocity-limited fixture, which is a virtual fixture limiting themaximum velocity of the manipulator.

FIG. 13 is a block diagram of architecture 1300 for rendering adaptivevirtual fixtures, in accordance with an example embodiment. A computingdevice, such computing device 1120 executing software 1310 can managethe virtual fixtures in a virtual environment based on data from aremote environment. Software 1310 can process and generate the virtualfixtures in real-time based on data in live sensor feed 1332 from videoand range sensor package 1330 in the remote environment. Sensor package1330 can be mounted on a device in the remote environment; e.g., aboarda remote manipulator or robot.

Software 1310 can render virtual fixtures implicitly defined by data inlive sensor feed 1332. That is, software 1310 can track and adaptvirtual fixtures to changes in the task space; e.g., as tasks are added,completed, updated, and removed; and to changes in the remoteenvironment; e.g., as objects move or change shape. Software 1330 cancommunicate position commands 1342 from human operator 1340 and/or plans1322 from high-level planner 1320 to remote manipulator 1350; e.g., asdesired position instructions 1354.

Remote manipulator 1350 can report actual position information 1352 tosoftware 1310. Software 1310 can receive actual position information1352 and data from live sensor feed 1332 to generate feedback 1344 tohuman operator 1340 about virtual fixtures and perhaps real objects inthe remote environment. As software 1310 determines tasks are completedand/or determines changes in the remote environment from data in livesensor feed 1332, software 1310 can provide changes 1324 to plansmaintained by high level planner 1320. Further, software 1310 canautonomously provide desired position information 1354 in somecircumstances; e.g., positions to be reached while following a guidancefeature, to avoid a collision, and/or in response to reaching forbiddenregion virtual fixture.

Table 2 below illustrates how virtual fixtures can be used insemi-autonomous mode (with both a human and an automated control loop)and in fully autonomous operation. In the latter, system intent is stillcommunicated to the operator using our haptic rendering algorithms,thereby providing an intuitive method for the operator to supervise (butnot control) the task. Conversely, the virtual fixtures can be used tocommunicate the outcome of the operation in manual/human control.

TABLE 2 Level of Automation Role of Virtual Fixtures Fully autonomousDuring fully autonomous operation, virtual fixtures operated byoperation software 1310 can communicate the intent of theautonomously-operating system to the human operator. The intent of thesystem can be shown with visual cues/displays (e.g., highlighting targetposition(s) in the workspace), auditory signal(s), and/or hapticfeedback. Semi-autonomous During semi-autonomous operation, the systemcan use the operation experience and cognition of the human operatorlinked with the precise spatial awareness of the computer system.Software 1310 can let the operator make decisions within ranges ofboundaries determined by likelihood estimates. The human operator canchange the ranges of boundaries to effectively modify the level ofautomation. Controlling ranges of boundaries for decision gives thehuman operator an intuitive control to the level of automation duringsemi-autonomous operation. The human operator can make higher-leveldecisions; i.e. ability to change a planned path within the range ofboundaries, or to intervene to avoid impending difficulties. Virtualfixtures can reside on a lower level such that a positioning task alwayswill be made exactly, and such that unintended collisions with themanipulator are prevented, mainly using force feedback. Manual operationDuring manual operation, all manipulation tasks can be done manually;e.g., without virtual fixtures. Instead of completely turning off thevirtual fixtures, the virtual fixtures can operate in the backgroundwithout sending feedback to the human operator. Virtual fixtures cancommunicate the outcome of the manual operator commands to thetemporarily deactivated autonomous system. By keeping the autonomoussystem up to date, the amount of time taken during switchover frommanual to (semi-) autonomous operation can be reduced.

Operator 1340 and/or software 1310 can vary the level of automation. Forexample, simpler manipulation tasks may be controlled by software 1310in the fully autonomous mode with operator 1340 providing positioncommands 1342 that reflect high-level decisions. If an unusual orexceptional event occurs (hence complexity increases), operator 1340 canreduce the level of automation. Combining the experience and skill of ahuman operator with the spatial awareness of the manipulation system,along with an adjustable autonomy framework, can both increaseefficiency and reliability of operations and allow execution morecomplex tasks than would otherwise be feasible with traditional manualor autonomous solutions.

A complex operation can be planned using standardized virtual fixtures;e.g., the operation can be specified as a sequence or other set oftasks, with each task shaped by virtual fixtures. When defined this way,both the human operator and the autonomous system will be able toreadily interpret this plan. Additionally, since the autonomous systemalso tracks the outcome of each task, procedural compliance can beobserved. If important steps are skipped, the human operator can benotified and/or other actions taken; e.g., remote actuators can be putinto hibernation or a reduced-performance mode until the correct task isbegun. This information can further be used to generate log files thatare sparse representations of already completed tasks. Log files canlater be used to generate performance statistics and maintenanceschedules.

The system can support flexible planning, allowing for some steps to beloosely defined (a necessary feature when operating in un-structured ordynamic environments). For instance, the system might be aware that alever needs to be pulled, but not which one. When the uncertain part ofthe plan is reached, the human operator will be asked to guide thesystem to the right lever using a point-and-click interface.

To best perform the task, the human operator should not perceive thatthe virtual fixtures are taking over completion the task and that thehuman is just riding along with an automated system. If the humanoperator is not sufficiently engaged, the human operator might be overlypassive and perhaps reduce focus and performance. For forbidden-regionvirtual fixtures, preventing passivity is straightforward since forcesare only sent when an operator is attempting entry into a forbiddenregion. That is, haptic feedback can be reasonably expected once theoperator realizes the interaction with the forbidden region.

For a guidance virtual fixture, however, the human operator may notalways anticipate the activation of a guidance force. Thus, virtualfixtures can use auditory and visual feedback, in addition to haptic toincrease the operator's situational awareness during manipulation tasks.This multi-sensory feedback can give the human operator a clearindication when a guidance force is about to be activated. For instance,visual virtual fixture(s) can provide visual indications and/or auditoryvirtual fixture(s) can emit sounds to indicate regions in task space inwhich the guidance force will be activated.

On-the-fly definition of virtual fixtures can enhance the high-levelarchitecture of planned operations constructed by standardized virtualfixtures. For this, a combined range and video sensor can be used in thesystem. The sensor can provide data at a first rate; e.g., at least 30Hz, and based on the sensor data, force feedback can be generated at asecond, faster rate; e.g., at least 1000 Hz. The system can utilizeprocessors, such as general-purpose graphics processing (GPGPU) hardwareand a customized parallelized low-level GPU, to process the sensor dataat least the first rate and generate force feedback at the second rate.

FIG. 14 is a block diagram of pipeline 1400 for parallel processing andgeneration of virtual fixtures, in accordance with an exampleembodiment. Pipeline 1400 can begin at block 1410, where range and videoimage data can be captured. For example, a computing device carrying outoperations of pipeline 1400 can receive one or more depth image at block1410, use cameras and/or sensors to obtain range and video image data,or otherwise obtain range (depth) and video image data.

At block 1420, a point cloud can be generated from the range and videoimage data. The point cloud can be a collection of points in a frame ofreference, such as a collection of Cartesian points in athree-dimensional space where the points correspond to objects whoseimages are captured at block 1410. In some embodiments, other datastructures, such as depth images, can be utilized and/or generated atblocks 1410 and 1420.

At block 1430, a bilateral filter can be applied. Bilateral filters canbe used to smooth data within the point cloud by weighted a valueassociated with point; e.g., a depth value, with a weighted average ofvalues from nearby points. For example, the points can be weighted usinga Gaussian kernel as a function of Euclidean distances in a Cartesianframe. In some embodiments, block 1430 can be omitted. In otherembodiments, other filters can be applied instead of or along with thebilateral filter.

In some embodiments, normal values can be determined for points in thepoint cloud. For example, pipeline 1400 can be utilized to carry outpart or the entire example haptic interaction algorithm for 6-DOF hapticrendering discussed above in the context of Table 1.

At block 1440, the image obtained at block 1410 can be segmented. Thatis, each pixel in the video image can be assigned a label. Each labelcan represent visual characteristics; e.g., a range of colors, alocation on a boundary (or within the interior) of an object, etc. Twopixels assigned to the same label have the same visual characteristics.

At block 1450, objects can be recognized from the segmented image. Insome embodiments, recognized objects can be associated with collectionsof points within the point cloud. For example, suppose an apple isrecognized in the segmented image and that the apple is depicted in aset of pixels SP of the segmented image. Let SPt be the set of points inthe point cloud associated with pixels SP. Since SP is associated withthe apple, then SPt can be associated with the apple as well.

At block 1460, virtual fixtures can be defined. The virtual fixtures canbe defined with respect to the object(s) recognized at block 1450. Tocontinue the example from block 1450, a virtual fixture VFa can beassociated with SP and SPt, and thus be associated with the recognizedapple. Then, if the apple moves in a later image captured at block 1410,the apple can be re-recognized in the later image at block 1450, andvirtual fixture VFa reassociated with the moved apple. Thus, virtualfixtures can be dynamically associated at block 1460 with objects in anenvironment captured in the range and video image data obtained at block1410 and recognized at block 1450.

In other embodiments, the virtual fixtures can be specified in terms ofgeometry in a remote environment and/or in a virtual environment. Forexample, a virtual object can be introduced into the virtualenvironment; e.g., a virtual cube or sphere. Then, a virtual fixture canbe associated with the virtual object. As another example, the virtualfixture can be defined with respect to a geometric region, volume, orother entity in either the remote or virtual environment without respectto another object. Other examples of pipeline 1400 are possible as well.

In some scenarios, the captured range and video image data isinsufficient; e.g., in cases of significant measurement noise or sensorocclusion. Measurement noise may be suppressed using filtering such asdiscussed above with respect to block 1430, To account for sensorocclusion, occlusion by a remote manipulator can be predicted based ondetailed models of the remote manipulator, kinematic equations, and dataabout manipulator positioning. Then, sensor data can be rejected thatcorrespond to regions predicted to be occluded by the remotemanipulator. In the rejected regions, sensor data can be interpolatedusing previously captured data. Further, by predicting a position of theremote manipulator for occlusion, the predicted position can also beused to verify the state of the manipulator during runtime, and soenhance operational safety.

FIG. 15 depicts an environment 1500 with a robot arm and multiplevirtual fixtures, including forbidden-region fixtures (FRFs) 1530, 1532and guidance fixtures (GFs) 1534, 1536 in accordance with an exampleembodiment. Environment 1500 is a virtual environment containingrepresentations of real objects, such as robot arm 1510 and toxicmaterial container, and representations of virtual objects, such asvirtual fixtures 1530, 1532, 1534, 1536, 1538.

In some embodiments, some or all of virtual fixtures 1530, 1532, 1534,1536, 1538 can be associated with a task list. Table 3 below shows anexample task list related to environment 1500 for robot arm 1510, wherethe task list is associated with virtual fixtures 1530, 1532, 1534,1536, 1538 as also shown in Table 3.

TABLE 3 Task List FRF 1530 FRF 1532 GF 1534 GF 1536 GF 1538 1. Go toValve Initially FRF. Initially FRF. Use GF to Initially Initially 1524Enable Enable move guide away guide entry/exit entry/exit toward fromvalve away from based on valid based on valid valve 1524 1526 materialauthorization authorization 1522 2. Obtain If valid Authorization toauthorization Enter Region provided 1532 (during task), deactivate FRF3. Turn Valve Upon Upon 1524 completion completion of task, of task,change to change to guide away guide from valve toward valve 1524 15264. Go to Valve Use GF to 1526 move toward valve 1526 5. Turn Valve Upon1526 completion of task, change to guide away from valve 1526 6. OpenDoor Door Handle/path to open door may be controlled by guidance fixturenot 1528 shown in FIG. 15. This fixture may be activated by completionof Task 5 and/or deactivated by completion of Task 6. 7. Enter AreaPathway to area 1534 can be provided by guidance fixture not Upon 1530shown in FIG. 15. This fixture may be activated by completion completionof Task 6. of task, change to guide toward material 1522 8. Go to ToxicUse GF to Material 1522 move toward material 1522 9. Get Toxic UponMaterial 1522 completion of task, GF can change to guide away frommaterial 1522 10. Remove Toxic Use GF to Material from move Region 1532away from material 1522. Other GFs can be activated. 11. Wait forTransport from outside Region 1530 12. Put Toxic Guidance features notshown in FIG. 15 can guide robot arm 1510 toward Material on transport.In some embodiments, part of guidance feature 1530 can be Transportdeactivated to let robot arm 1510 and/or transport work together. 13.Close Door Guidance features not shown in FIG. 15 may guide robot arm1510 to door 1528 and to close door (similar to those for Task 6). 14.Leave Deactivate Activate FRF Perimeter Region FRF until once robot arm1530 robot arm 1510 leaves region leaves region 1532 1530.

Each of virtual fixtures 1530, 1532, 1534, 1536, 1538, is shown in FIG.15 associated with a number. For example, virtual fixtures 1530 and 1532have associated respective indicators 1550 and 1552 showing respectivenumbers 2 and 12, while each of virtual fixtures 1534, 1536, 1538 isshown displaying respective numbers 1, 4, and 8. In the environmentshown in FIG. 15, where tasks in the task list of Table 3 have not yetbegun, each number correspond to the next task that the virtual fixtureis associated with. So, virtual fixture 1534 is associated with Task 1of the task list, virtual fixture 1532 is associated with Task 2, and soon. The indicator can be updated as tasks complete; e.g., uponcompletion of Task 1, the virtual fixture indicator for virtual fixture1534 can be updated to refer to Task 3, which is the succeeding task forthat virtual fixture. In some embodiments, an indicator of a virtualfixture can indicate multiple tasks associated with the virtual fixture;e.g., the next N tasks, with N>1, or all tasks referring to the virtualfixture. In other embodiments, upon completion of the last taskinvolving a virtual fixture, an indicator for the virtual fixture canchange to “Complete” or a similar indication of completion. In stillother embodiments, upon completion of the last task involving a virtualfixture, the virtual fixture can be removed from environment 1500.

In another scenario, consider a manipulation task where two valves needto be turned, such as valves 1524 and 1526. In this scenario, the orderin which the valves are turned is irrelevant. The human operator can seevalves 1524, 1526 on a display with both highlighted as valid targets.In addition to the valves the operator also sees virtual cones; e.g.,guidance fixtures 1534 and 1536, extending out from each handle. Sincethe computer system providing environment 1500 cannot be sure whichvalve the human operator will approach first, no force feedback isgiven. Instead, if the operator commands the manipulator into the coneextending out valve 1524, force feedback can be automatically activatedto guide the manipulator into a perfect configuration for a valve turn.That is, the human operator makes the high level decision and thecomputer system makes sure that it is being executed with high accuracy.To the operator the whole procedure will feel effortless, just as if themanipulator ‘just happened’ to slide into the right configuration byaccident. Similar user interfaces exist in many document processors,where the user drags a figure to roughly the right position in thedocument and the software automatically aligns it with the margins.

Virtual fixtures can be programmatically controlled. For example,task-management software can control one or more associated virtualfixtures. In some embodiments, the task-management software can be partof software 1310 discussed above in the context of FIG. 13.

The task-management software can include software that manages tasks ona task list with associated virtual fixtures such as shown in Table 3.The task-management software can allow the operator to define a tasklist, define virtual fixtures as needed for the task list, define taskson the task list, and then aid the operator and associated remotedevices; e.g., robot arm 1510 of FIG. 15, to carry out the tasks. Thetask-management software can ensure that tasks on the task list arecarried out in order. In some embodiments, a task can be defined interms of multiple operators acting in parallel.

The task-management software can use the task list in combination withreal-time sensor data to automatically transition between the differentvirtual fixtures, such as the transition between grasping and turning avalve. In cases where multiple choices are possible, the task-managementsoftware will have the ability to let the operator decide while insemi-autonomous levels of automation. In cases when no operator ispresent and/or in a fully autonomous level of operation, thetask-management software can use a number of techniques to make adecision, such as conditional probabilities and maximum likelihoodvotes. The task-management software can avoid undesirable transientsthat can arise from virtual fixture transitions as well as from taskstate changes by careful matching of terminal and initial conditions ofsuccessive control epochs as well as enforcement of protectiveconstraints on control states and outputs.

The task-management software, in addition to serving as an interface andcontrol system, can constantly monitor operations toward task completionand continually tracking of operation outcomes with associatedlikelihoods. The task-management software can adaptively generate andtransition between virtual fixtures identified for a given task.Although the task-management software can execute the transitionsbetween virtual fixtures autonomously, autonomous operation is notalways desirable. Rather, a level of automation can be modified, eitherby the computer system or by the operator as discussed above in thecontext of FIG. 13 and Table 2.

As another example, some or all virtual fixtures in a virtualenvironment can be associated with a script or other software thatcontrols part or all of the operation of the virtual fixture. The scriptcan include instructions, such as but not limited to instructions thatcan operate based on conditions of the virtual fixture; e.g., a type ofvirtual fixture (forbidden-region fixture, guidance fixture), conditionsin the environment; e.g., time, temperature, weather conditions, watercurrents, wind condition, chemical conditions, biological conditions,nuclear conditions, sensory input related to conditions in theenvironment, task lists and/or tasks, and/or other conditions. Theinstructions can change the virtual fixture, perhaps based on acondition in the environment.

For example, a forbidden-region fixture that allows entry to a forbiddenregion to a device D if proper authorization is provided by D at time ofentry can have a virtual-fixture script such as shown in Table 4 below.

TABLE 4 ID = myID( ); send_authentication_request( ); R =get_authentication_response( ); If (valid_entry(ID, R)) then  Open(ID);

The virtual-fixture script can be invoked when the entry to theforbidden region associated with the virtual fixture is sought, when anobject comes in contact with the virtual fixture, or at some other time.The virtual-fixture script first assigns an identifier “ID” to thevirtual fixture using a myID( ) function—in some embodiments, this canbe performed prior to script execution. Then, an authentication requestcan be sent and a response “R” to the authentication request received.The virtual-fixture script then determines if R provides valid entry tothe virtual fixture using the valid_entry( ) function—if R does providevalid entry to the forbidden region associated with the virtual fixture,then the virtual fixture is opened using the open( ) function.Virtual-fixture scripts can have other characteristics; e.g., avirtual-fixture script can be written in other computer languages, canutilize many other functions, can be compiled and/or interpreted, and/orcan perform other operations. Many virtual-fixture scripts are possibleas well.

FIG. 16 is a flowchart of method 1600, in accordance with an exampleembodiment. Method 1600 can begin at block 1610, where a computingdevice can receive first depth data about an environment. At block 1620,the computing device can generate a first plurality of points from thefirst depth data. At block 1630, the computing device can determine ahaptic interface point.

At block 1640, the computing device can define a virtual fixture for theenvironment. In some embodiments, wherein determining the virtualfixture for the environment can include: determining one or more objectsin the environment from the first depth data; determining a first objectof the one or more objects, the object having a first location in theenvironment; and associating the virtual fixture with the first objectand the first location.

In other embodiments, the virtual fixture is at least one fixtureselected from the group of fixtures consisting of a forbidden-regionfixture and a guidance fixture. In particular of the other embodiments,the virtual fixture can include a forbidden-region fixture. Then,determining the first force vector can include: determining an initialforce vector between the HIP and the first plurality of points;determining whether the HIP is within a forbidden region defined by theforbidden-region fixture; and in response to determining that the HIP iswithin the forbidden region: generating a forbidden-region-force vectoraway from the forbidden region, and determining the first force vectorbased on the initial force vector and the forbidden-region-force vector.

In other particular of the other embodiments, the virtual fixture caninclude a guidance fixture. Then, determining the first force vectorcomprises: determining an initial force vector between the HIP and thefirst plurality of points; determining whether the HIP is within aguidance region defined by the guidance fixture, wherein the guidanceregion comprises a target; and in response to determining that the HIPis within the guidance region: generating a guidance-region-force vectortoward the target; and determining the first force vector based on theinitial force vector and the guidance-region-force vector.

In still other embodiments, defining the virtual fixture for theenvironment can include: determining one or more instructions related tothe virtual fixture and defining the virtual fixture based on the one ormore instructions. In particular of the still other embodiments, the oneor more instructions can be configured to change the virtual fixturebased on a condition in the environment. In more particular of the stillother embodiments, the condition in the environment comprises acondition selected from the group of conditions consisting of a timecondition, a temperature condition, a weather condition, a water-currentcondition, a wind condition, a chemical condition, a biologicalcondition, and a nuclear condition.

At block 1650, the computing device can determine a first force vectorbetween the haptic interface point and the first plurality of points.The first force vector is based on the virtual fixture. In someembodiments, the virtual fixture can include a standoff fixture. Thestandoff fixture can define a target and a safety region. The safetyregion can be associated with the target. Then, determining the firstforce vector can include: determining an initial force vector betweenthe HIP and the first plurality of points; determining whether the HIPis within the safety region; and in response to determining that the HIPis within the safety region: generating a safety-region-force vectoraway from the target, and determining the first force vector based onthe initial force vector and the safety-region-force vector.

In other embodiments, the HIP can have a velocity. The virtual fixturecan include a velocity-limited fixture. The velocity-limited fixture candefine a velocity-limited region and a maximum velocity. Then,determining the first force vector comprises: determining an initialforce vector between the HIP and the first plurality of points;determining whether the HIP is within the velocity-limited region; inresponse to determining that the HIP is within the velocity-limitedregion: determining whether the velocity of the HIP exceeds the maximumvelocity, and in response to the velocity of the HIP exceeding themaximum velocity, determining a velocity-limiting vector opposing thevelocity of the HIP, and determining the first force vector based on theinitial force vector and the velocity-limiting vector.

At block 1660, the computing device can send a first indication ofhaptic feedback based on the first force vector. In some embodiments,method 1600 can further include: receiving second depth data about theenvironment at the computing device, where the second depth data differsfrom the first depth data; determining one or more objects in theenvironment from the second depth data; determining whether the one ormore objects includes the first object; and in response to determiningthat the one or more objects include the first object: determining asecond location in the environment for the first object, and associatingthe virtual fixture with the first object and the second location.

In other embodiments, the environment can include a virtual toolassociated with the HIP. The virtual tool can be defined in terms of atool-translation component and a tool-rotation component. The target canbe configured to be defined in terms of a target-translation componentand a target-rotation component. Then, method 1600 can further include:in response to determining that the HIP is within the guidance region,aligning the tool-rotation component with the target-rotation component.

In still other embodiments, method 1600 can further include: determininga task list that can be associated with a plurality of tasks to beperformed in the environment, where the task list comprises a firsttask. Then, determining the virtual fixture can include determining thevirtual fixture based on the task list. In particular of theseembodiments, determining the virtual fixture based on the task list caninclude: determining the virtual fixture initially to be a first virtualfixture; determining whether the first task is completed; and inresponse to determining that the first task is completed, changing thefirst virtual fixture to a second virtual fixture, wherein the firstvirtual fixture and the second virtual fixture differ.

In some particular of these embodiments, the first virtual fixture canbe a guidance fixture and the second virtual fixture can be aforbidden-region fixture. In other particular of these embodiments, thefirst virtual fixture can be a forbidden-region fixture and wherein thesecond virtual fixture can be a guidance fixture. In other particular ofthese embodiments, the task list can further include a second task. Thevirtual fixture can include a first virtual fixture having a first typeof virtual fixture. Then, determining the virtual fixture based on thetask list can include: determining whether the first task is completed;in response to determining that the first task is completed, changingthe first type of virtual fixture to a second type of virtual fixture,wherein the first type of virtual fixture and the second type of virtualfixture differ; determining whether the second task is completed; and inresponse to determining that the second task is completed, changing thesecond type of virtual fixture to a third type of virtual fixture,wherein the second type of virtual fixture and the third type of virtualfixture differ.

In some of the other particular of these embodiments, the first type ofvirtual fixture can be a forbidden-region type of virtual fixture, thesecond type of virtual fixture can be a guidance-region type of virtualfixture, and wherein the third type of virtual fixture can be theforbidden-region type of virtual fixture.

Haptic Rendering Related to Underwater Tasks

The herein-described techniques for 3DOF and 6DOF dynamic hapticrendering can be utilized to carry out tasks underwater. Operations inthe deep sea, or hazardous marine environments, typically requiremanipulation from remotely operated vehicles (ROVs). For example, hapticrendering can be used in the context of an omnidirectional mobile robotwith underwater grippers; e.g., robot 1000 discussed above in thecontext of at least FIG. 10. These underwater tasks can include tasksfor search and rescue, underwater infrastructure maintenance, repair,mining, military operations, and waste cleanup.

Subsea manipulation is a challenging task, and one that is necessary ina number of applications. Examples include the construction of cabledocean observatories, biological studies of hydrothermal vent colonies,oil and gas operations, subsea mining, and other activities that requireprecise movement on the seafloor. In these applications, a humanoperator's perception is important, especially in the presence ofdisturbances due to currents, low visibility situations, or otherscenarios that require adaptation. Another difficulty in performingmanipulation tasks with remote robots arises in making precise motionsand interactions in the presence of dynamically changing environments.These include objects or structures that must be avoided or protected.These objects can be moving, and possibly changing in size. For example,in underwater manipulation for repair (e.g., oil platforms) or forbiological sampling, precision manipulation is required despitedisturbances from water flow and turbulence.

Underwater tasks can involve manipulation in the vicinity of sensitiveor delicate structures. Visual features, such as scaling and tremordampening can improve performance. However, operator error,communication delay and limited visual feedback can cause a robot to beaccidentally driven into sensitive structures and result in irreparabledamage. The problem is further amplified in the absence of hapticfeedback to the operator. If the contact is not visually detected by theoperator, a resulting collision can cause serious damage.

Consider, for example, the common task of mating electrical connectionsunderwater. In principle this is a simple maneuver; however, operatorsoften find it difficult to successfully mate connections usingunderwater manipulators, such as robot arms including the ARM 5E MINIfive-function electric manipulator configured for subsea use. At best,this difficulty results in excessively long duration operations thatdrive up the cost of subsea operations. At worst, hardware is destroyed,often requiring expensive replacements. Further, operational costs forROVs can be expensive; e.g., $1 per second or more. Thus, there is greatincentive to maximize the efficiency and capability of human/co-robotmanipulation tasks in subsea environments.

An underwater depth-enabled camera can be used provide depth images for3DOF and 6DOF haptic rendering to carry out underwater tasks. The depthimages can provide haptic feedback to a human operator, giving with asense of touch along with a visual representation objects seen by thecamera. For example, a human operator can use a tele-operated roboticdevice equipped with the underwater depth-enabled camera to directrobotic removal of ordnance from lake, river or sea bottoms. An addedfeature of the herein-described techniques is prevention of robot endeffector contact with undesirable locations through imposition ofvirtual fixtures, such as forbidden-region fixtures.

In some embodiments, the depth-enabled camera can be used in combinationwith a metal detector or metal detector array. This combination ofdevices can provide a human tele-operator with information about bothdepth (distance to target) and metal profile(s) within the area ofoperation. The metal detector (array) can be mounted on an underwatercraft near manipulator(s), such as robotic arms, to allow 2D or 3D metalprofiles to be mapped without using a visual or acoustical detector. Anarray of small metal detectors can be configured in a cross pattern,arranged around the camera system to allow 2D metal profiling data to betaken simultaneously with the image capture. The metal profile will begenerated from the array moving across the area while scanning. Analternate metal detector design is to obtain the metal profile systemusing a phase array design where radiation pattern of the B field can beshifted by adjusting the phase difference between driving coils insideeach metal sensor.

In an example system, a depth-enabled camera and metal detector arraycamera optimized for reconnaissance and surveying can be placed in a SeaRay-like ROV operable from a surface craft, such as a barge. The ROV canbe linked to the surface craft by a tether, ensuring ROV recovery aswell as uninterruptible power and data lines. The ROV can be equippedwith additional video cameras and lights in addition to thedepth-enabled camera and metal detector array. The ROV can provide thedata necessary to construct a complete 3D picture of what is ahead ofthe surface craft and update a 3D underwater map in real time.

Providing haptic feedback from underwater sensors to an operator at thesurface has the potential to transform subsea manipulation capability.Virtual fixtures will allow manipulators to precisely maneuver insensitive environments where contact should not be made with surroundingobjects or structures. Biological studies of animal colonies aroundhydrothermal vents could benefit greatly from such a capability—allowingscientists to carefully gather data with unprecedented resolution andproximity to sensitive organisms. Common tasks such as connector matingbetween instruments can be carried out very efficiently by creatingguidance fixture near male and female connectors. Thus, time, expense,and equipment can be saved, and the environment preserved using theherein-described haptic feedback techniques to perform underwater tasks.

FIG. 17 is a block diagram of system 1700, in accordance with an exampleembodiment. System 1700 is shown with respect to three environments:proximate physical environment 1702, remote virtual environment 1704,and representing remote physical environment 1706. As one of manypossible examples, remote physical environment 1706 can be an underwaterenvironment.

In proximate physical environment 1702, master console 1710 can be, orinclude, a computing device configured to enable a human operator (notshown in FIG. 17) to interact with remote physical environment 1706. Forexample, master console 1710 can have a computing device, hapticinterface device, and display configured to provide visualizations, suchas computing device 320 b, haptic interface device 316, display 312, andvisualization 318 discussed above with respect to FIG. 3A.

Master console 1710 can communicate with network 1720. FIG. 1700 showsthat master console 1710 can receive feedback from network 1720, such asvisual feedback (VF) 1712 and haptic force feedback 1714 (F_(H)), andcan send commands, such as commands with a haptic device, or instructed,end effector position 1716 (P_(H)) of a remotely-controlled device inremote physical environment 1706, such as robot 1784.

Network 1720 can connect computer devices in proximate physicalenvironment 1702 and remote physical environment 1706; e.g., connectmaster console 1710 with computing device 1720. For example, network1720 can have some or all of the functionality of network 310 of FIG.3A. FIG. 17 illustrates that network 1720 can communicate data,including but not limited to, visual feedback 1712, haptic forcefeedback 1714, and haptic device position 1716, between proximatephysical environment 1702 and remote physical environment 1706.

Remote virtual environment 1704 can include computing device(s) 1724configured to communicate with network 1720 and remote physicalenvironment 1706. For example, computing device(s) 1724 can have some orall of the functionality of computing device 320 a of FIG. 3A. FIG. 17indicates that computing device(s) 1724 can include visualization module1730, virtual fixture module 1732, inverse kinematics module 1734,registration module 1750, forward kinematics module 1752, and controllermodule 1754. Each or all of modules 1730, 1732, 1734, 1750, 1752, 1754can be or comprise software executable on processor(s) of computingdevice(s) 1724.

FIG. 17 shows that remote physical environment 1706 can include sensors,such as camera 1780 and depth sensor 1782, and robot 1784. For examples,camera 1780 can be or include one or more of depth enabled cameras 346a, 346 b, 346 c, 346 d shown in FIG. 3A and robot 1784 can be or includeremote controlled device 336 also shown in FIG. 3A. In some embodiments,camera 1780 can include depth sensor 1782 and be configured forunderwater use; e.g., camera 1800 discussed below in the context ofFIGS. 18A and 18B.

In operation, camera 1780 can capture light within remote physicalenvironment 1706 and generate images 1760. At the same time, depthsensor 1782 can capture a 3D depth map of remote physical environment1706 in the form of point cloud 1782 obtain depth information about andgenerate point cloud (PC) 1762. Robot 1784 can obtain data about actualrobot joint angle(s) for actuator(s), effector(s), robot arm(s), and/orother component(s) of robot 1784. The data about actual robot jointangle(s) is shown in FIG. 17 as robot joint angle (A) 1764. Images 1760,point cloud 1762, and robot joint angle 1764 can be communicated tocomputing device 1724 in remote virtual environment 1704.

More specifically, FIG. 17 shows that images 1760 are communicated toregistration module 1750 and visualization module 1730, point cloud 1762is communicated to registration module 1750, and robot joint angle 1764is communicated to forward kinematics module 1752 and controller module1754.

For point cloud 1762, point cloud 1762 can be registered in the robotframe to aid processing within system 1700. FIG. 17 shows thatregistration module 1750 registers point cloud 1762 to generateregistered point cloud (RPC) 1742. To register point cloud 1762,registration module 1750 can generate a transformation between pointcloud 1762 and a point of view of robot 1784 based on image(s) 1760.Registered point cloud 1742 can then be communicated from registrationmodule 1750 to visualization module 1730 and virtual fixture module1732.

Visualization module 1730 can use visual data from image(s) 1712 anddepth data from registered point cloud 1742 to generate visual feedback1712. Visual feedback 1712 can also include visual information relatedto haptic force feedback 1714; i.e., if haptic force feedback 1714indicates a force due to collision with an object, then correspondingvisual feedback; e.g., a visual collision alert or other indication ofcollision, can be added to visual feedback 1712. Similarly, end effectorposition commands, such as end effector position 1716, can be used invisual feedback; e.g., if end effector position 1716 indicates that anend effector moves in a direction D, then visual feedback 1712 can showcorresponding movement of (a visualization of) an end effector of robot1784.

Forward kinematics module 1752 can convert robot joint angle 1764 to endeffector coordinates (P) 1744. Virtual fixture module 1732 can employregistered point cloud 1742, end effector coordinates 1744, and endeffector position 1716 to compute and provide haptic force feedback 1714to the operator via master console 1710 and network 1720. Virtualfixture module 1732 also can compute and communicate desired endeffector position (P_(D)) 1740 to inverse kinematics module 1734.Inverse kinematics module 1734 can convert desired end effector position1740 to desired joint angle(s) (θ_(D)) 1746, which in turn can beconverted into motor torque commands (τ) 1766 by controller module 1754.Upon reception of motor torque commands 1766, robot 1784 can movemotor(s) in accord with the received commands 1766. In some embodiments,robot 1784 can have multiple controllable components; e.g., havemultiple arms and/or end effectors. In still other embodiments, robot1784 can be mobile and move in accord with commands provided from masterconsole 1710. In these embodiments, motor torque commands 1766 can begenerated to move multiple components of robot 1784 and/or to move robot1784 itself. In still other embodiments, system 1700 can use multiplecameras 1780, depth sensors 1784, and/or robots 1784 in remote physicalenvironment 1706, which can be controlled by one or more operators atmaster console 1710 and using information provided by computingdevice(s). Other configurations of system 1700 are possible.

Depth Cameras for Underwater Virtual Fixtures

FIG. 18A is a block diagram of camera 1800 configured to provide imagesfor underwater haptic rendering, in accordance with an exampleembodiment. Camera 1800 is a video camera built for underwater use andconfigured to capture depth data and light and provide the captureddelight. The captured depth data and light can be provided as images,such as depth images that can be used to generate haptic feedback andvirtual fixtures. In some embodiments, camera 1800 can provide frames atVGA resolution (720×900 pixels) at a frame rate of 30 frames/second inRGB plus Depth (RGBD) format. Camera 1800 can determine depthdisplacement based on variations of the geometry of a projectedpseudo-random pattern of near-infrared (NIR) light. Generally, visiblelight can be light with wavelength(s) in a visible-light range between400 and 700 nanometers (nm), and near-infrared light can be light withwavelength(s) in a near-infrared range between 700 and 2000 nm.

FIG. 18A shows that camera 1800 includes near-infrared light source1810, diffraction grating 1812, infrared camera 1814, visible lightsource 1820, polarizer 1822, visible light camera 1824, processor 1830,and communication interface 1840. Near-infrared light source 1810 cangenerate infrared light using one or more light-emitting diodes (LEDs),such as an array of 2 W laser diodes having a wavelength (λ) of 830 nm,and a change in wavelength (Δλ) of 3 nm, such as laser diodes made byJDSU.

Camera 1800 can rely on a structured light principle where thedisplacement of an object is determined by variations of the geometry ofa projected pattern. For example, infrared light source 1810 anddiffraction grating 1812 can be used to project a pseudo-random patternhaving varying intensities. Camera 1800 can also determine an expectedpseudo-random pattern based on the projected pseudo-random pattern, andthen determine variations in geometry by comparing the captured andexpected pseudo-random patterns.

Camera 1800 can incorporate a near-infrared monochrome sensor (e.g.Microsoft/X853750001/VCA379C7130, 1200×1600 pixels), perhaps as part ofinfrared camera 1814, for measuring depth from the distortion betweenthe projected and observed pseudo-random patterns. For reduce the backscattering, a pulse duration of infrared light source 1810 can be setmuch shorter than the time expected for emitted infrared light to travelto the target. Infrared camera 1814 can capture light in thenear-infrared range and generate image(s) using the captured light,including but not limited to images of projected pseudo-random patternsof near-infrared light emitted by infrared light source 1810 anddiffraction grating 1812. In some embodiments, infrared camera cangenerate images having a predetermined depth resolution or range ofdepth resolutions; e.g., a depth resolution of 1.5-2.5 mm.

Visible light source 1820 can use one or more light sources, such asvisible light LEDs, to generate visible light. In some embodiments,visible light source 1820 can have blue-green filters and/or blue-greenLEDs to generate visible light in the 450 to 480 nm range. For example,visible light source 1820 can be, or include, a Sea-View SV-Q10K 500WHalogen Unit. Visible light in 450-480 nm range can match a blue-greentransmission window of water, thus improving the contrast and range ofoperation. Polarizer 1822 can reduce glare and undesirable reflectionsfrom the environment. Polarizer 1822 also can detect changes in apolarization state of light scattered by water and suspended particles,allowing for enhanced contrast. Visible light camera 1814 can capturelight in the visible light range and generate image(s) using thecaptured light.

Images generated by infrared camera 1814 and visible camera 1824 can bereceived by processor 1830 and provided to communication interface 1840.For example, processor 1830 can interleave depth-related images receivedfrom infrared camera 1814 with visible light images from visible lightcamera 1824 into pairs of images, where each pair of images includes adepth-related image and a visible-light image captured at substantiallythe same time; e.g., within 1/30th of a second of each other.

In some embodiments, processor 1830 can alternate activation of lightsources 1810, 1820 at a rate based on a frame rate of camera 1800. Forexample, suppose camera 1800 is configured to generate N video framesper second, with N>1, and let a frame speed be X seconds, where X≦1/N.Then, light source 1810 can be activated to emit near-infrared light for0.5× seconds and a depth-related image captured by near-infrared camera1814 using the emitted near-infrared light, then light source 1820 canbe activated to emit visible light for 0.5× seconds and a visible-lightimage captured by visible light camera 1824 using the emitted visiblelight. In particular of these embodiments, a single camera configured tocapture light in both the visible-light range and near-infrared rangecan perform the functionality of infrared camera 1814 and visible lightcamera 1824 by capturing both images of the pair of images within theframe speed of X seconds.

In other embodiments, the pair of images includes a depth-related imageand a visible light image captured; e.g., processor 1830 can activateboth light sources 1810 and 1820 at the same time and both infraredcamera 1814 and visible light camera 1824 can capture light emitted atthe same time in the respective near-infrared and visible-light ranges.Thus, each pair of images can provide depth and visible lightinformation about an environment captured at substantially the sametime.

FIG. 18B is an exploded view of camera 1800, in accordance with anexample embodiment. FIG. 18B shows camera 1800 with infrared lightsource 1810, infrared camera 1814, visible light source 1820, visiblelight camera 1824, communication interface 1840, watertight body 1850,and light shield 1852. Watertight body 1850 can hold components ofcamera 1800 on and/or inside of body 1850. Light shield 1852 can shieldcameras 1814, 1824 from direct light emitted from light sources 1810,1820, and so reduce glare on images taken by cameras 1814, 1824.

For underwater applications, camera 1800 can be waterproofed. Camera1800 can be enclosed in watertight body 1850 configured to isolatecamera 1800 from the harm of water using materials that have little orno impact on optical performance. Camera 1800 can include water-tightconnectors that allow external communication via optical fibersconnected to communication interface 1840. In some embodiments not shownin FIG. 18A, camera 1800 can include a battery configured for poweringcamera 1800. In some embodiments, one or more adaptors can enableconnection of camera 1800 to a borescope. The adapter(s) can combinenear-infrared images and visible-light images, and recover the depthinformation from the near-infrared images using a wavelength-specificbeam splitter.

FIG. 18C depicts a sensor system 1860, in accordance with an exampleembodiment. Sensor system 1860 includes metal detector(s) 1870 and ascanning camera system, where the scanning camera system has a laserscanner 1862 and line scan camera 1864. Metal detectors, such as metaldetector(s) 1870, are discussed below in more detail in the context ofat least FIGS. 24A through 27B.

In the example shown in FIG. 18C, sensor system 1860 is scanning inregion 1880 along a scan direction 1882 upward and rightward. Laserscanner 1862 generates an illumination field using laser beams such aslaser beam 1872—FIG. 18C show laser beams generated by laser scanner1862 using dashed and dotted lines. Laser scanner 1862 can use usinghigh power collimated laser lights to generate collimated laser beamswith minimal cross sections, such as laser beam 1872. FIG. 18C showsthat laser scanner 1860 is configured to emit laser beams, such as laserbeam 1872 along scan lines, such as scan lines 1874 a, 1874 b, 1874 c,1874 d, and 1874 e, within region 1880.

Sensor system 1860 allows small individual elements of a target scene,such as a portion of region 1880 swept out by scan lines 1874 a-1874 e,to be illuminated one at a time with minimal backscatter. For example,FIG. 18C shows a field of view 1876 for line scan camera 1864 as arelatively small portion of region 1880. At the same time, a narrowfield of view photodetector (not shown in FIG. 18C) can track field ofview 1876 and reduce forward scatter from light reflected from thetarget scene.

FIG. 19 shows near-infrared image 1900 of object 1910 in clear water, inaccordance with an example embodiment. To capture image 1900, anear-infrared camera and light source, such as near-infrared camera 1814and near-infrared light source 1810, were suspended about 85 cm (about33.5 inches) above a flat surface. A container with clear water about 20cm (about 7.9 inches) deep was placed on the flat surface below thenear-infrared camera and light source. The near-infrared camera wasaimed at the body of water. Then, the near-infrared light sourcegenerated a pseudo-random pattern and a near-infrared image, now image1900, was captured by the near-infrared camera. The pseudo-randompattern is observable in FIG. 19 as dots scattered throughout image1900; e.g., dots visible below and to either side of a white circle usedto outline object 1900.

FIG. 20A is near-infrared image 2000 of an object in murky water, inaccordance with an example embodiment. Image 2000 was captured using thesame setup used for FIG. 19—a near-infrared camera and light source,such as near-infrared camera 1814 and near-infrared light source 1810,were suspended about 85 cm above a flat surface, about 20 cm of waterwas placed on a container above the flat surface, and the near-infraredcamera aimed at the water. Then, as with image 1900, a pseudo-randompattern of near-infrared light was projected onto the water and anear-infrared image, now image 2000, was captured. Object 2010 a ofimage 2000 is the same object as object 1910 of image 1900.

However, for image 2000, murky water was used rather than the clearwater used for image 1900 of FIG. 19. FIG. 20A indicates that only partof the projected pattern makes it through the body of murky water.

FIG. 20B is image 2050 representing depth data related to image 2000 ofFIG. 20A, in accordance with an example embodiment. The depth data wasobtained by processing the partial projected pattern obtained from image2050. The depth data can be distorted due to the index differencebetween air and water, but as long as the depth data can be obtainedthrough a body of water, it can be corrected by softwarepost-processing. Image 2050 shows object 2010 b clearly, indicating thatvalid depth data for object 2010 b was obtained even though only apartial projected pattern on object 2010 a was obtainable from image2010.

FIG. 21 shows graph 2100 of light absorption in water versus lightwavelength and graph 2150 of light absorption in water versus lightwavelength over a range of water temperatures, in accordance with anexample embodiment. Graph 2100, shown in the upper portion of FIG. 21,shows that the water absorption spectrum has a minimum near the NIRwavelength, indicating that the NIR wavelength is suitable forunderwater. Graph 2150, shown in the lower portion of FIG. 21, showswater absorption for light in a range of wavelengths between 700 and 900nm as a function of temperatures ranging from 30° C. to 40° C. witharrows 2160 and 2162 indicating a direction of temperature.

Resolving Dot Pattern Interference in Multiple Structured Light Cameras

The simultaneous use of multiple depth-enabled cameras; e.g., camera1800 can extend coverage regions for haptic rendering, overcomeocclusions, and enable complete full-circle captures of environments. Insome cases, structured light sensing can degrade when multipledepth-enabled cameras are pointing at the same scene, due to continuous,non-modulated projection of pseudo-random patterns; e.g., as discussedabove in the context of diffraction grating 1812 of FIG. 18A. Thecontinuous, non-modulated projection of pseudo-random patterns can causecrosstalk when dot patterns of devices interfere with one another.

Herein are provided techniques and devices that can mitigate structuredlight crosstalk. In some embodiments, these techniques and devices canbe utilized without modification of the internal electronics, firmware,or software of a depth-enabled camera and without degradation of thedepth-enabled camera's frame rate.

One technique to eliminate crosstalk involves time multiplexing where anear-infrared light source associated with each camera in a system ofcameras can be modulated and synchronized at a predetermined modulatedtime frequency.

FIG. 22 is a block diagram of time-multiplexed system 2200 ofnear-infrared cameras 2224, 2234 configured to capture images ofenvironment 2212, in accordance with an example embodiment. Timed switch2210 can generate two periodic signals, T1 signal 2214 and T2 signal2216, with both signals having the same period but only one signal isactive at any specified time. For example, timed switch 2210 can usegenerate a square wave with equal active/inactive periods for T1 signal2214 and an inverted version of the square wave for T2 signal 2216 (orvice versa). Other techniques for generating T1 signal 2214 and T2signal 2216 can be used as well.

Then, when T1 signal 2214 is active, near-infrared light source 2220 canbe activated. When activated, near-infrared light source 2220 can shinenear-infrared light 2226 through diffraction grating 2222 and ontoenvironment 2212. Subsequently, near-infrared camera 2224 can capturenear-infrared light 2228 reflected from environment 2212 as an image ofT1 images 2240. Near-infrared camera 2224 can place T1 images 2240 ontoconnector 2260.

When T2 signal 2216 is active, near-infrared light source 2230 can beactivated. When activated, near-infrared light source 2230 can shinenear-infrared light 2236 through diffraction grating 2232 and ontoenvironment 2212. Subsequently, near-infrared camera 2234 can capturenear-infrared light 2238 reflected from environment 2212 as an image ofT2 images 2250. Near-infrared camera 2224 can place T2 images 2250 ontoconnector 2260 for use by to device(s) outside of system 2200. As T1images 2240 are placed onto connector 2260 at times based on T1 signal2214 and T2 images 2250 are placed onto connector 2260 at times based onT2 signal 2216, T1 images and T2 images can be interleaved in time asinterleaved T1/T2 images 2262. As shown in FIG. 22, interleaved T1/T2images 2262 can be sent from system 2200, perhaps for use by device(s)outside system 2200.

In other embodiments not shown in FIG. 22, more than two near-infraredcameras can be used. In these embodiments, each of N near-infraredcameras, N>2, can be assigned an equal, periodic, time interval; e.g., aperiodic time slot. During an assigned time slot, a corresponding lightsource to the near-infrared camera can be illuminated, image(s) ofenvironment 2212 can be captured by the near-infrared camera and theimage(s) transmitted. Timeslots can be allocated in round-robin fashionto each of the N near-infrared cameras as long as images of environment2212 are to be captured and transmitted.

In some embodiments, near-infrared cameras 2224, 2234 are activecontinuously. When a signal for corresponding camera is not active;e.g., T1 signal 2214 for NIR camera 2224, and T2 signal 2216 fornear-infrared camera 2234, images generated by that camera can beconsidered to be invalid and so not be processed. To reduce any imageprocessing applicable to interleaved T1/T2 images 2262, a shuttersynchronized with T1 and T2 signals can be used to block or allow lightfrom light sources 2220, 2230 so a corresponding camera captures lightfrom a light source associated with the corresponding camera; e.g.,light from light source 2220 is blocked by the shutter when T1 signal2214 is not active and is allowed by the shutter when T1 signal 2214 isactive.

Another technique to eliminate crosstalk involves is wavelength-divisionmultiplexing where each near-infrared light source associated with acamera in a system of cameras has a different wavelength.

FIG. 23 is a block diagram of frequency-multiplexed system 2300 ofnear-infrared cameras 2324, 2334 configured to capture images ofenvironment 2310, in accordance with an example embodiment. λ1 lightsource 2320 can be configured to emit light in a sub-range of thenear-infrared range of frequencies, where the sub-range includes awavelength λ1. Similarly, λ2 light source 2330 can be configured to emitlight in a sub-range of the near-infrared range of frequencies, wherethe sub-range includes a wavelength λ2, where the sub-range includingwavelength λ1 does not overlap the sub-range including wavelength λ2.For example, if λ1=820 nm and λ1=900 nm, the sub-range including λ1 canbe 800-840 nm, and the sub-range including λ2 can be 880-920 nm. In someembodiments, the sub-range can include light of one frequency; e.g.,using the previous example, the sub-range including λ1 can have onefrequency of 820 nm and the sub-range including λ2 can have onefrequency of 900 nm. Many other example values of λ1, λ2, and sub-rangesassociated with λ1 and λ2 are possible as well.

λ1 light source 2320 can emit λ1 light 2326 that reaches environment2310. Environment 2310 can reflect some or all of the light includingfrequency λ1 as λ1 light 2328 to near-infrared camera 2324.Subsequently, near-infrared camera 2324 can capture near λ1 light 2328as an image of λ1 images 2340. Near-infrared camera 2324 can place λ1images 2340 onto connector 2360.

λ1 light source 2323 can emit λ2 light 2336 that reaches environment2310. Environment 2310 can reflect some or all of the light includingfrequency λ2 as λ2 light 2338 to near-infrared camera 2334.Subsequently, near-infrared camera 2334 can capture near λ1 light 2328as an image of λ1 images 2350. Near-infrared camera 2324 can place λ2images 2350 onto connector 2360.

As shown in FIG. 23, λ1 images and λ2 images can be sent from system2300 as λ1 and λ2 images 2362, perhaps for use by device(s) outsidesystem 2200. A device receiving λ1 and λ2 images 2362 can use filter(s)to ensure that images taken in light having a corresponding wavelengthis detected and analyzed.

In other embodiments not shown in FIG. 23, more than two near-infraredcameras can be used. In these embodiments, each of N near-infraredcameras, N>2, can be assigned a non-overlapping frequency, and perhaps anon-overlapping sub-range of frequencies that includes the assignedfrequency, within the near-infrared range of frequencies. Each cameracan then have an associated light source configured to emit light withthe assigned non-overlapping frequency (or emit light in the assignedsub-range of frequencies, if sub-ranges are assigned). The camera cancapture light reflected from environment 2310 having the assignednon-overlapping frequency as an image, and send any captured images.

Metal Detecting Systems

A metal detector can work alongside a camera system, such as camera1800, system 2200, or system 2300. For example, sensor system 1860discussed above in the context of FIG. 18C includes both a camera andmetal detector(s). An array of the metal detectors can be arrangedaround the camera system to allow simultaneous capture of 2D metalprofiling data and images. The array of metal detectors can be arrangedin a pattern; e.g., the array can be arranged in a cross pattern. Inother scenarios, a metal detector or an array of metal detectors can bemounted on a vessel or near actuators, such as robotic arms, to map 2Dor 3D metal profiles without using a visual or acoustical detector.

The metal profile, such as but not limited to a line scan, raster scan,or graph of magnetic flux over time, can be generated by an array ofmetal detectors moving across a target region while scanning. In someembodiments, the metal profile detector can utilize a phase array designwhere radiation pattern of the B field can be shifted by adjusting thephase difference between driving coils inside each metal sensor.

To reliably detect smaller metal items underwater; e.g., smallmunitions, a compact metal detector can be used. In some embodiments,the metal detector can have a relatively-small sensing resolution; e.g.,a resolution of one square centimeter, and be capable of differentiatingmetal of different shapes and geometry using a fiber-optic sensor.

A metal detector can be based on a metal sensor having magnetostrictivematerials. A magnetostrictive material is a material that changes shapeor dimensions in the presence of a magnetic field. The metal detectoroperates by comparing a light signal sent through a loop of opticalfiber coated with a magnetostrictive material with a light signal sentthrough an uncoated reference optical fiber. When the coated loop passesover para- or diamagnetic material, the magnetic field can induce strainin the fiber through magnetostriction of the coating. The induced straincan change the index of refraction within the coated fiber through thephotoelastic effect, which changes the coated fiber's response relativeto the reference. For example, the strain can causes a change in thelength of the optical path. The metal detector can include aninterferometer to measure the phase changes induced in the strainedfiber relative to light passing through the reference coil. The presenceof such phase changes indicates that metal has been detected.

Metal detectors using magnetostrictive materials can be configured to berelatively small in size. Further, magnetostrictive-based metaldetectors can identify shapes or topographic features of metal targetswhile operating as a single metal detector or a metal detector in anarray of metal detectors.

FIG. 24A shows structure 2400 of a metal sensor, in accordance with anexample embodiment. Structure 2400 includes a layer of magnetostrictivematerial 2410 a, a light carrying material 2412, and another layer ofmagnetostrictive material 2410 b. For example, the light carryingmaterial can be an optical fiber and the magnetostrictive material canbe a ferromagnetic polymer. Structure 2400 can be used when layers ofmagnetostrictive material 2410 a and 2410 b coat light carryingmaterial.

FIG. 24B is a diagram of interferometer-based metal detector 2420, inaccordance with an example embodiment. Laser 2422 can send a lightsignal to optical coupler 2430, which is connected to both sensing coil2440 and reference coil 2442. In metal detector 2420, each of coils 2440and 2442 is configured to act as a sensing arm in an interferometer.Both sensing coil 2440 and reference coil 2442 include a light pathway;e.g., one or more optical fibers in the coil. However, sensing coil 2440includes fiber coated with magnetostrictive material, while referencecoil 2442 includes fiber that is not coated with magnetostrictivematerial; that is, coil 2440 includes structure 2400 while coil 2442does not include structure 2400.

After light passes through each of sensing coil 2440 and reference coil2442, the light is provided to optical coupler 2432 which can pass thelight from sensing coil 2440 to detector 2452 and passes the light fromreference coil 2440 to detector 2450. Light reaching detector 2450 canbe compared to light reaching detector 2452; e.g., to determine phasechanges in light indicating differences in sensing coil 2440 andreference coil 2442. That is, metal detector 2420 includes aMach-Zehnder interferometer for detecting changes in light between coils2440 and 2442 that represent the presence or absence of metal.

FIG. 24C is a diagram of interferometer-based metal detector 2460, inaccordance with an example embodiment. Laser 2462 can send a lightsignal to optical coupler 2470, which is connected to detector 2472,sensing coil 2480, and reference coil 2480. Both sensing coil 2480 andreference coil 2482 include a light pathway; e.g., one or more opticalfibers in the coil. In metal detector 2460, each of coils 2480 and 2482is configured to act as a sensing arm in an interferometer. However,sensing coil 2480 includes fiber coated with magnetostrictive material,while reference coil 2482 includes fiber that is not coated withmagnetostrictive material; that is, coil 2480 includes structure 2400while coil 2482 does not include structure 2400.

Light passing through sensing coil 2480 can be provided to mirror 2490,which can reflect the provided light back through sensing coil 2480 andcoupler 2470 to detector 2472. Similarly, light passing throughreference coil 2482 can be provided to mirror 2492, which can reflectthe provided light back through reference coil 2482 and coupler 2470 todetector 2472. Light reaching detector 2472 from sensing coil 2480 canbe compared to light reaching detector 2472 from reference coil 2482e.g., to determine phase changes in light indicating differences insensing coil 2480 and reference coil 2482. That is, metal detector 2460includes a Michelson interferometer for detecting changes in lightbetween coils 2440 and 2442 that represent the presence or absence ofmetal.

FIG. 25A is a diagram of metal detection system 2500 that includesmultiple metal detectors 2510 a, 2510 b, 2510 c, 2510 d, 2510 e, 2510 f,2510 g, 2510 h, and 2510 i, in accordance with an example embodiment.More specifically, metal detection system 2500 includes nine metaldetectors 2510 a, 2510 b, 2510 c, 2510 d, 2510 e, 2510 f, 2510 g, 2510h, and 2510 i arranged in a cross-shaped pattern. Each of metaldetectors 2510 a, 2510 b, 2510 c, 2510 d, 2510 e, 2510 f, 2510 g, 2510h, and 2510 i can be an interferometer-based metal detector, such asmetal detector 2420 discussed above in more detail in the context ofFIG. 24B or metal detector 2460 discussed above in more detail in thecontext of FIG. 24C.

In particular, each of metal detectors 2510 a, 2510 b, 2510 c, 2510 d,2510 e, 2510 f, 2510 g, 2510 h, and 2510 i can utilize magnetostrictivematerial as part of the interferometer-based metal detector. When one ormore metal detectors 2510 a-2510 i are on an area having paramagneticand/or diamagnetic material or move over an area having paramagneticand/or diamagnetic material, the one or more metal detectors 2510 a-2510i can detect a perturbation due to presence of a paramagnetic and/ordiamagnetic material in the field. Based on a sequence of magnitude ofthe perturbation collected by the array of sensors, we can reconstructthe metal profile of the area the sensors pass over. The metal profilecan include graphs, diagrams, line scans, and/or raster scans discussedbelow.

Other patterns of metal detectors than cross-shaped patterns can be usedin metal detection systems. FIG. 25B is a diagram of metal detectionsystem 2520 that includes multiple metal detectors, in accordance withan example embodiment. More specifically, metal detection system 2520includes nine metal detectors 2530 a, 2530 b, 2530 c, 2530 d, 2530 e,2530 f, 2530 g, 2530 h, and 2530 i arranged in an X-shaped pattern. Eachof metal detectors 2530 a, 2530 b, 2530 c, 2530 d, 2530 e, 2530 f, 2530g, 2530 h, and 2530 i can be an interferometer-based metal detector,such as metal detectors 2510 a-2510 i discussed above in the context ofFIG. 25A.

FIG. 25C is a diagram of metal detection system 2540 that includesmultiple metal detectors, in accordance with an example embodiment. Morespecifically, metal detection system 2520 includes twenty-five metaldetectors arranged in a grid-based pattern. Each of the twenty-fivemetal detectors, including metal detector 2550, are shown in FIG. 25C asmarked with a “MD”. Each metal detector in metal detection system 2540can be an interferometer-based metal detector, such as metal detectors2510 a-2510 i discussed above in the context of FIG. 25A. Other patternsof metal detectors and other metal detectors can be used in metaldetection systems as well.

FIG. 26A depicts a sensor system using metal detection system 2500, inaccordance with an example embodiment. Sensor system 2600 can generate ametal profile, such as a line scan, by moving an array of metaldetectors 2510 a-2510 i in metal detection system 2500 along scan lines2620 while the metal detectors are performing metal scans; e.g.,searching for metallic objects below sensor system 2600. FIG. 26A showsmetal detector(s) of metal detection system 2500 performing metal scan2610 while sensor system 2600 is in motion along an upper-most scan lineof scan lines 2620.

FIG. 26B depicts sensor system 2630 using metal detection system 2500,in accordance with an example embodiment. Sensor system 2630 can utilizea phase array design where radiation pattern of the B field 2640 can beshifted by adjusting a phase difference between driving coils insideeach metal detector 2510 a-2510 i of metal detector system 2500 whilemetal detector 2510 a-2510 i are performing metal scans. A raster scancan be generated by adding phase differences in metal scans between thecross-pattern of metal detectors in metal detector system 2500. Togenerate the raster scan, sensor system 2640 can travel along rasterscan path 2650, which is a zigzag shaped path as shown in FIG. 26B. Thegenerated raster scan can be utilized as a metal profile for objectsalong raster scan path 2650.

In some embodiments, sensor system 2600 and/or sensor system 2630 canutilize more and/or different metal detection systems than metaldetection system 2500; e.g., one or more metal detection systems 2520and/or metal detection systems 2540 can be utilized as well as, orrather than metal detection system 2500 as shown in FIGS. 26A and 26B.Other techniques to generate metal profiles by using sensor systemsequipped with metal detection systems are possible as well.

FIG. 27A is image 2710 of example metal objects 2702, 2704, 2706, inaccordance with an example embodiment. Image 2710 shows an arrangementof metal objects, shown in left-to-right order, as rod 2702 runningalong a left-most side of image 2700, C clamp 2704 centrally placed inimage 2700 and rod 2706 running along the left-most side of image 2700.

FIG. 27B is a graph 2720 of an output signal from aninterferometer-based metal detector while scanning for metal objects inaccordance with an example embodiment. Graph 2720 shows magnetic fluxdensity in Gauss versus time as observed by an interferometer-basedmetal detector that is part of an array of metal detectors in a metaldetection system; e.g., metal detector 2510 a of metal detection system2500. When the magnetic flux density reaches a peak, the metal detectorhas located a metal object.

Graph 2710 also includes inset 2730 with a copy of image 2710 includingrod 2702, C Clamp 2704, and rod 2706, in accordance with an exampleembodiment. In the scan performed to generate graph 2720, the metalobjects in image 2710 were scanned starting 0.2 after the beginning ofthe scan and the metal detector traveled in the direction of scandirection (SD) 2732 with respect to inset 2730.

Graph 2720 shows data for an interferometer based metal detector as anupper/darker grey line of graph 2720 and data for a Hall effect sensoras lower/lighter grey line of graph 2720. The in interferometer basedmetal detector and the Hall effect sensor were at the same position inmetal detection system 2710. Graph 2720 shows that both sensors hadsimilar responses; e.g., both the upper and lower lines of graph 2720have similar peaks and valleys during the scan.

Specifically, at a time about 0.2 seconds as indicated by arrow 2740,the metal detector passed over rod 2702. Then, at a time at about 0.25seconds as indicated by arrow 2742, the metal detector passed over aleftmost prong of C Clamp 2704 as shown in inset 2730 i.e., the top ofthe “C”, and at a time at about 0.38 seconds as indicated by arrow 2744,the metal detector passed over a rightmost prong of C Clamp 2704; i.e.,the bottom of the “C”. Later, at a time of about 0.45 seconds asindicated by arrow 2746, the metal detector passed over rod 2706.

Graph 2720 shows that the metal detector indicated a peak in magneticflux, and thus detected metal, when passing over rod 2702. The magneticflux fell off from the peak of about 1200 Gauss at 0.2 seconds withinapproximately 0.01 seconds indicating that a narrow band of metal lie inthe path of the metal detector while traveling along scan direction2732. A similar peak is noted at about 0.45 seconds when the metaldetector passed over rod 2706. Peaks were reached at about 0.25 secondswhen the leftmost prong of C Clamp 2704 was detected and at about 0.38seconds when the rightmost prong of C Clamp 2704, but in between thesetwo peaks, the metal detector dropped to a relatively-low Gauss level of1000 Gauss (or less) at about 0.30 to 0.32 seconds, indicating that themetal detector did not detect metal during that interval; i.e., themetal detector did not detect the body of C Clamp 2704 according tograph 2720.

FIG. 27C is a diagram 2750 of metal objects determined based on the datain FIG. 27B, in accordance with an example embodiment. Diagram 2750 ofFIG. 27C is a 2D reconstructed map of rod 2702, C Clamp 2704, and rod2706 shown in image 2710 of FIG. 27A using data from all metal detectorsin a metal detection system while passing over the metal objectsdepicted in image 2710. Diagram 2750 shows each of rod 2752, C clamp2754, and rod 2706 in the same relative positions as rod 2702, C Clamp2704, and rod 2706 are shown in image 2710. Further, the general shapeof rod 2702 and C Clamp 2704 of image 2710 are respectively reflected asrod 2752 and C Clamp 2754 of diagram 2750. Additionally, rod 2756 indiagram 2750 accurately depicts both the relative position and shape ofrod 2706 shown in image 2710.

FIG. 28 illustrates a scenario 2800 where vessel 2810 detects andretrieves underwater objects 2820, 2822, in accordance with an exampleembodiment. In scenario 2800, vessel 2810 is floating in river 2830above river bed 2832. In other scenarios, vessel 2810 can be used in anocean, lake, pond, or other water environment than river 2830. Vessel2810 has robot arms and hands acting as actuators 2840 and 2850. Eachactuator 2840, 2850 can be partially or completely controlled by aremote operator; e.g., a human operator aboard vessel 2810 or located inanother location. In some scenarios, the remote operator can adjust alevel of automation of actuator 2840 and/or 2850; e.g., to be manuallyoperated, semi-autonomously operated, or be fully autonomous operated.

The remote operator can receive feedback about the operation of actuator2840 (and/or 2850) and/or about the environment using sensor system 2842(and/or 2852). Sensor system 2842 (and/or 2852) can include one or moredepth-enabled cameras including visible-light camera(s) andnear-infrared camera(s), metal detectors/metal detection systems, andlight sources including visible-light light source(s) and near-infraredlight source(s) perhaps configured to emit pseudo-random patterns oflight. In particular, sensor system 2842 (and/or 2852) can be configuredto provide depth images, where the depth images can be used to generatehaptic feedback for three or more degrees of freedom in controllingactuator 2840 (and/or 2850).

In scenario 2800, the remote operator of actuator 2840 (and/or 2850)operates actuator 2840 (and/or 2850) in accord with a task list, wherethe task list is configured to control virtual fixtures associated withtasks of retrieving object 2820 from river bed 2832 and object 2822 fromunder river bed 2832. Once an object is retrieved, the object can beplaced in object container 2860; e.g., using a guidance fixtureassociated with the task list configured to guide actuator 2840 (and/or2850) from a location of the object to a location of object container2860. Other virtual fixtures, such as but not limited toforbidden-region fixtures, can be associated with the task list.

In scenario 2800, another remote operator can use remotely-operatedvehicle 2862 to provide additional information regarding the environmentincluding objects 2820, 2822, actuators 2840, 2850, river bed 2832,and/or river 2830. Remotely-operated vehicle 2862 can be attached tovessel 281 via tether 2864. Tether 2864 can include power and/orcommunication cables to enable remotely-operated vehicle 2862 to bepowered from and/or communicate with vessel 2810. In other embodiments,remotely-operated vehicle 2862 can operate without tether 2864.

FIG. 29 is a flowchart of method 2900, in accordance with an embodiment.Method 2900 can be carried out with a depth-enabled camera, such ascamera 1800. Method 2900 can start at block 2910, where a camera, suchas camera 1800 discussed above in the context of at least FIGS. 18A and18B, can capture, within a predetermined interval of time, first lightwithin a first frequency range of light in the underwater environmentand second light within a second frequency range of light in theunderwater environment. In some embodiments, the first frequency rangeof light can include a range of frequencies of light between 450nanometers (nm) and 480 nm. Then, the second frequency range of light offrequencies can include a range of frequencies of light between 825 and835 nm.

At block 2920, the camera can generate a first image based on the firstlight and a second image based on the second light. The first frequencyrange of light can differ from the second frequency range of light.

At block 2930, the camera can send at least the first image and thesecond image. In some embodiments, sending at least the first image andthe second image from the camera can include: generating a predeterminednumber of frames per second, where each frame includes a first frameimage based on light within the first frequency range received at aspecified time and a second frame image based on light within the secondfrequency range captured at the specified time, and sending thepredetermined number of frames per second from the camera. In particularof these embodiments, the predetermined number of frames per second canbe a number N, with N>1. Then, the predetermined interval of time can beless than or equal to 1/N seconds.

In some embodiments, the camera can include a first light emitting diode(LED) and a second LED. Then, method 2900 can further include:generating light within the first frequency range of light using thefirst LED and generating light within the second frequency range oflight using the second LED. In particular of these embodiments, thecamera further includes diffraction grating. Then, generating lightwithin the second frequency range of light can include the scatteringthe light generated by the second LED into a pseudo-random pattern oflight within the second frequency range using the diffraction grating.In more particular of these embodiments, method 2900 can further includecapturing the pseudo-random pattern of light within the second frequencyrange using the camera, and determining distortion within the secondfrequency range of light by comparing the captured pseudo-random patternof light to an expected pseudo-random pattern of light.

In other embodiments, the camera can include a polarizer. Then, method2900 can further include reducing reflections in at least one image ofthe first image and the second image using the polarizer. In still otherembodiments, the camera can include an optical adaptor. Then, method2900 can further include separating light received at the device intothe first light and the second light using the optical adaptor. In yetother embodiments, the first image and second image are configured to beutilized for generating haptic feedback.

The above detailed description describes various features and functionsof the disclosed systems, devices, and methods with reference to theaccompanying figures. In the figures, similar symbols typically identifysimilar components, unless context dictates otherwise. The illustrativeembodiments described in the detailed description, figures, and claimsare not meant to be limiting. Other embodiments can be utilized, andother changes can be made, without departing from the spirit or scope ofthe subject matter presented herein. It will be readily understood thatthe aspects of the present disclosure, as generally described herein,and illustrated in the figures, can be arranged, substituted, combined,separated, and designed in a wide variety of different configurations,all of which are explicitly contemplated herein.

With respect to any or all of the ladder diagrams, scenarios, and flowcharts in the figures and as discussed herein, each block and/orcommunication may represent a processing of information and/or atransmission of information in accordance with example embodiments.Alternative embodiments are included within the scope of these exampleembodiments. In these alternative embodiments, for example, functionsdescribed as blocks, transmissions, communications, requests, responses,and/or messages may be executed out of order from that shown ordiscussed, including substantially concurrent or in reverse order,depending on the functionality involved. Further, more or fewer blocksand/or functions may be used with any of the ladder diagrams, scenarios,and flow charts discussed herein, and these ladder diagrams, scenarios,and flow charts may be combined with one another, in part or in whole.

A block that represents a processing of information may correspond tocircuitry that can be configured to perform the specific logicalfunctions of a herein-described method or technique. Alternatively oradditionally, a block that represents a processing of information maycorrespond to a module, a segment, or a portion of program code(including related data). The program code may include one or moreinstructions executable by a processor for implementing specific logicalfunctions or actions in the method or technique. The program code and/orrelated data may be stored on any type of computer readable medium suchas a storage device including a disk or hard drive or other storagemedium.

The computer readable medium may also include physical and/ornon-transitory computer readable media such as computer-readable mediathat stores data for short periods of time like register memory,processor cache, and random access memory (RAM). The computer readablemedia may also include physical and/or non-transitory computer readablemedia that stores program code and/or data for longer periods of time,such as secondary or persistent long term storage, like read only memory(ROM), optical or magnetic disks, compact-disc read only memory(CD-ROM), for example. The computer readable media may also be any othervolatile or non-volatile storage systems. A computer readable medium maybe considered a computer readable storage medium, for example, or atangible storage device.

The terms physical and/or tangible computer-readable medium and physicaland/or tangible computer-readable media refer to any physical and/ortangible medium that can be configured to store instructions forexecution by a processor, processing unit and/or computing device. Sucha medium or media can take many forms, including but not limited to,non-volatile media and volatile media. Non-volatile media includes, forexample, read only memory (ROM), flash memory, magnetic-disk memory,optical-disk memory, removable-disk memory, magnetic-tape memory, harddrive devices, compact disc ROMs (CD-ROMs), direct video disc ROMs(DVD-ROMs), computer diskettes, and/or paper cards. Volatile mediainclude dynamic memory, such as main memory, cache memory, and/or randomaccess memory (RAM). Many other types of tangible computer-readablemedia are possible as well. As such, the herein-described data storagecan comprise and/or be one or more physical and/or tangiblecomputer-readable media.

Moreover, a block that represents one or more information transmissionsmay correspond to information transmissions between software and/orhardware modules in the same physical device. However, other informationtransmissions may be between software modules and/or hardware modules indifferent physical devices.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

1. A device configured for operation in an underwater environment,comprising: a camera configured to capture, within a predeterminedinterval of time, first light within a first frequency range of light inthe underwater environment and second light within a second frequencyrange of light in the underwater environment, wherein the camera isconfigured to generate a first image based on the first light and asecond image based on the second light, wherein the first frequencyrange of light differs from the second frequency range of light, andwherein the camera comprises a communication interface configured atleast to send at least the first image and the second image and toreceive one or more commands based on haptic feedback with the hapticfeedback generated based on the first image and the second image.
 2. Thedevice of claim 1, wherein the camera comprises a first light emittingdiode (LED) configured to generate light within the first frequencyrange of light and a second LED configured to generate light within thesecond frequency range of light.
 3. The device of claim 2, wherein thecamera further comprises a diffraction grating configured to scatterlight from the second LED into a pseudo-random pattern of light withinthe second frequency range.
 4. The device of claim 3, wherein the secondlight comprises the pseudo-random pattern of light within the secondfrequency range, and wherein the device is configured to determinedistortion within the second frequency range of light by comparing thepseudo-random pattern of light captured within the second light to anexpected pseudo-random pattern of light.
 5. The device of claim 1,wherein the camera comprises a polarizer configured to reducereflections at least one image of the first image and the second image.6. The device of claim 1, further comprising an optical adaptorconfigured to separate light received at the device into the first lightand the second light.
 7. The device of claim 1, wherein the camera isconfigured to generate a predetermined number of frames per second,wherein each frame comprises a first frame image based on received lightwithin the first frequency range at a specified time and a second frameimage based on received light within the second frequency range at thespecified time, and wherein the communication interface is configured totransmit the predetermined number of frames per second.
 8. The device ofclaim 7, wherein the predetermined number of frames is a number Ngreater than one, and wherein the predetermined interval of time is lessthan or equal to 1/N seconds.
 9. The device of claim 1, wherein thecamera is configured to be waterproof.
 10. The device of claim 1,wherein the haptic feedback is based on visual information of the firstimage and on depth information determined from the second image.
 11. Thedevice of claim 1, further comprising an actuator configured to becontrolled by the one or more commands.
 12. A method, comprising:capturing, within a predetermined interval of time, first light within afirst frequency range of light in an underwater environment and secondlight within a second frequency range of light in the underwaterenvironment using a camera; generating a first image based on the firstlight and a second image based on the second light using the camera,wherein the first frequency range of light differs from the secondfrequency range of light; and sending at least the first image and thesecond image from the camera.
 13. The method of claim 12, wherein thecamera comprises a first light emitting diode (LED) and a second LED,and wherein the method further comprises: generating light within thefirst frequency range of light using the first LED; and generating lightwithin the second frequency range of light using the second LED.
 14. Themethod of claim 13, wherein the camera further comprises a diffractiongrating, and wherein generating light within the second frequency rangeof light comprises scattering the light generated by the second LED intoa pseudo-random pattern of light within the second frequency range usingthe diffraction grating.
 15. The method of claim 14, further comprising:capturing the pseudo-random pattern of light within the second frequencyrange using the camera; and determining distortion within the secondfrequency range of light by comparing the captured pseudo-random patternof light to an expected pseudo-random pattern of light.
 16. The methodof claim 12, wherein the camera comprises a polarizer, and wherein themethod further comprises: reducing reflections in at least one image ofthe first image and the second image using the polarizer.
 17. The methodof claim 12, wherein the camera comprises an optical adaptor, andwherein the method further comprises: separating light received at thedevice into the first light and the second light using the opticaladaptor.
 18. The method of claim 12, wherein sending at least the firstimage and the second image from the camera comprises: generating apredetermined number of frames per second, wherein each frame comprisesa first frame image based on light within the first frequency rangecaptured at a specified time and a second frame image based on lightwithin the second frequency range captured at the specified time; andsending the predetermined number of frames per second from the camera.19. The method of claim 18, wherein the predetermined number of framesis a number N greater than one, and wherein the predetermined intervalof time is less than or equal to 1/N seconds.
 20. The method of claim12, wherein the first image and second image are configured to beutilized for generating haptic feedback.